Note: This is test shard 1 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from AdminCliTest
[ RUN ] AdminCliTest.RebalancerReportOnly
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 21:25:24.746414 30025 test_util.cc:268] Using random seed: 1036196992
I20240502 21:25:24.758061 30025 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:25:24.758142 30025 ts_itest-base.cc:113] --------------
I20240502 21:25:24.758193 30025 ts_itest-base.cc:114] 5 tablet servers
I20240502 21:25:24.758260 30025 ts_itest-base.cc:115] 3 replicas per TS
I20240502 21:25:24.758328 30025 ts_itest-base.cc:116] --------------
2024-05-02T21:25:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:25:24Z Disabled control of system clock
I20240502 21:25:24.784099 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.82.126:40371
--webserver_interface=127.29.82.126
--webserver_port=0
--builtin_ntp_servers=127.29.82.84:32943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.82.126:40371 with env {}
W20240502 21:25:25.155990 30038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:25.156304 30038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:25.156443 30038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:25.164366 30038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 21:25:25.164479 30038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:25.164553 30038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 21:25:25.164638 30038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 21:25:25.173452 30038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:32943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.82.126:40371
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.82.126:40371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.29.82.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:25.174737 30038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:25.176597 30038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:25.187922 30043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:25.188613 30046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:25.189297 30044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:25.189960 30038 server_base.cc:1034] running on GCE node
I20240502 21:25:25.190703 30038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:25.192119 30038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:25.193387 30038 hybrid_clock.cc:648] HybridClock initialized: now 1714685125193314 us; error 38 us; skew 500 ppm
I20240502 21:25:25.193756 30038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:25.200387 30038 webserver.cc:460] Webserver started at http://127.29.82.126:41201/ using document root <none> and password file <none>
I20240502 21:25:25.201123 30038 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:25.201262 30038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:25.201591 30038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:25.204334 30038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "6902e668c6fe42a28364a0c462a40e20"
format_stamp: "Formatted at 2024-05-02 21:25:25 on dist-test-slave-xzsm"
I20240502 21:25:25.205067 30038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "6902e668c6fe42a28364a0c462a40e20"
format_stamp: "Formatted at 2024-05-02 21:25:25 on dist-test-slave-xzsm"
I20240502 21:25:25.210083 30038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 21:25:25.213357 30052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:25.214591 30038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20240502 21:25:25.214754 30038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal
uuid: "6902e668c6fe42a28364a0c462a40e20"
format_stamp: "Formatted at 2024-05-02 21:25:25 on dist-test-slave-xzsm"
I20240502 21:25:25.214941 30038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:25.236826 30038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:25.237583 30038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:25.237869 30038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:25.255791 30038 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.126:40371
I20240502 21:25:25.255831 30107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.126:40371 every 8 connection(s)
I20240502 21:25:25.257149 30038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 21:25:25.260201 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 30038
I20240502 21:25:25.260488 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 21:25:25.261346 30108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:25.272292 30108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6902e668c6fe42a28364a0c462a40e20: Bootstrap starting.
I20240502 21:25:25.275108 30108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6902e668c6fe42a28364a0c462a40e20: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:25.276120 30108 log.cc:824] T 00000000000000000000000000000000 P 6902e668c6fe42a28364a0c462a40e20: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:25.279349 30108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6902e668c6fe42a28364a0c462a40e20: No bootstrap required, opened a new log
I20240502 21:25:25.287884 30108 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6902e668c6fe42a28364a0c462a40e20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6902e668c6fe42a28364a0c462a40e20" member_type: VOTER last_known_addr { host: "127.29.82.126" port: 40371 } }
I20240502 21:25:25.288312 30108 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6902e668c6fe42a28364a0c462a40e20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:25.288434 30108 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6902e668c6fe42a28364a0c462a40e20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6902e668c6fe42a28364a0c462a40e20, State: Initialized, Role: FOLLOWER
I20240502 21:25:25.289108 30108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6902e668c6fe42a28364a0c462a40e20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6902e668c6fe42a28364a0c462a40e20" member_type: VOTER last_known_addr { host: "127.29.82.126" port: 40371 } }
I20240502 21:25:25.289336 30108 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6902e668c6fe42a28364a0c462a40e20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:25.289485 30108 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6902e668c6fe42a28364a0c462a40e20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:25.289665 30108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6902e668c6fe42a28364a0c462a40e20 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:25.291918 30108 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6902e668c6fe42a28364a0c462a40e20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6902e668c6fe42a28364a0c462a40e20" member_type: VOTER last_known_addr { host: "127.29.82.126" port: 40371 } }
I20240502 21:25:25.292379 30108 leader_election.cc:302] T 00000000000000000000000000000000 P 6902e668c6fe42a28364a0c462a40e20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6902e668c6fe42a28364a0c462a40e20; no voters:
I20240502 21:25:25.292992 30108 leader_election.cc:288] T 00000000000000000000000000000000 P 6902e668c6fe42a28364a0c462a40e20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:25.293162 30113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6902e668c6fe42a28364a0c462a40e20 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:25.294293 30113 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6902e668c6fe42a28364a0c462a40e20 [term 1 LEADER]: Becoming Leader. State: Replica: 6902e668c6fe42a28364a0c462a40e20, State: Running, Role: LEADER
I20240502 21:25:25.295500 30108 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6902e668c6fe42a28364a0c462a40e20 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:25:25.295881 30113 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6902e668c6fe42a28364a0c462a40e20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6902e668c6fe42a28364a0c462a40e20" member_type: VOTER last_known_addr { host: "127.29.82.126" port: 40371 } }
I20240502 21:25:25.298949 30115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6902e668c6fe42a28364a0c462a40e20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6902e668c6fe42a28364a0c462a40e20. Latest consensus state: current_term: 1 leader_uuid: "6902e668c6fe42a28364a0c462a40e20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6902e668c6fe42a28364a0c462a40e20" member_type: VOTER last_known_addr { host: "127.29.82.126" port: 40371 } } }
I20240502 21:25:25.299258 30115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6902e668c6fe42a28364a0c462a40e20 [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:25.299062 30114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6902e668c6fe42a28364a0c462a40e20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6902e668c6fe42a28364a0c462a40e20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6902e668c6fe42a28364a0c462a40e20" member_type: VOTER last_known_addr { host: "127.29.82.126" port: 40371 } } }
I20240502 21:25:25.299436 30114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6902e668c6fe42a28364a0c462a40e20 [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:25.301126 30121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:25:25.306622 30121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:25:25.316345 30121 catalog_manager.cc:1347] Generated new cluster ID: 44ca3235db174466b5f82cb3824ba9fe
I20240502 21:25:25.316474 30121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:25:25.327811 30121 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:25:25.329457 30121 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:25:25.346133 30121 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6902e668c6fe42a28364a0c462a40e20: Generated new TSK 0
I20240502 21:25:25.347064 30121 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:25:25.364742 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.65:0
--local_ip_for_outbound_sockets=127.29.82.65
--webserver_interface=127.29.82.65
--webserver_port=0
--tserver_master_addrs=127.29.82.126:40371
--builtin_ntp_servers=127.29.82.84:32943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240502 21:25:25.704804 30132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:25.705132 30132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:25.705231 30132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:25.713320 30132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:25.713508 30132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.65
I20240502 21:25:25.722898 30132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:32943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.29.82.65
--webserver_port=0
--tserver_master_addrs=127.29.82.126:40371
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.65
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:25.724334 30132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:25.726068 30132 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:25.738502 30137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:25.738973 30138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:25.739665 30140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:25.740561 30132 server_base.cc:1034] running on GCE node
I20240502 21:25:25.741192 30132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:25.742086 30132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:25.743304 30132 hybrid_clock.cc:648] HybridClock initialized: now 1714685125743247 us; error 42 us; skew 500 ppm
I20240502 21:25:25.743638 30132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:25.745779 30132 webserver.cc:460] Webserver started at http://127.29.82.65:40639/ using document root <none> and password file <none>
I20240502 21:25:25.746466 30132 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:25.746604 30132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:25.746907 30132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:25.749488 30132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6402e732f88a4bbea512b429a7e2e6ad"
format_stamp: "Formatted at 2024-05-02 21:25:25 on dist-test-slave-xzsm"
I20240502 21:25:25.750250 30132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6402e732f88a4bbea512b429a7e2e6ad"
format_stamp: "Formatted at 2024-05-02 21:25:25 on dist-test-slave-xzsm"
I20240502 21:25:25.755316 30132 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240502 21:25:25.758582 30146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:25.759920 30132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:25:25.760074 30132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6402e732f88a4bbea512b429a7e2e6ad"
format_stamp: "Formatted at 2024-05-02 21:25:25 on dist-test-slave-xzsm"
I20240502 21:25:25.760244 30132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:25.788375 30132 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:25.789286 30132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:25.789608 30132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:25.790560 30132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:25.792191 30132 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:25.792275 30132 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:25.792413 30132 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:25.792471 30132 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:25.825914 30132 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.65:39605
I20240502 21:25:25.825963 30261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.65:39605 every 8 connection(s)
I20240502 21:25:25.827226 30132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 21:25:25.832870 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 30132
I20240502 21:25:25.833112 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 21:25:25.836612 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.66:0
--local_ip_for_outbound_sockets=127.29.82.66
--webserver_interface=127.29.82.66
--webserver_port=0
--tserver_master_addrs=127.29.82.126:40371
--builtin_ntp_servers=127.29.82.84:32943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 21:25:25.841696 30263 heartbeater.cc:347] Connected to a master server at 127.29.82.126:40371
I20240502 21:25:25.842041 30263 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:25.842998 30263 heartbeater.cc:511] Master 127.29.82.126:40371 requested a full tablet report, sending...
I20240502 21:25:25.845016 30070 ts_manager.cc:194] Registered new tserver with Master: 6402e732f88a4bbea512b429a7e2e6ad (127.29.82.65:39605)
I20240502 21:25:25.846789 30070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.65:48139
W20240502 21:25:26.174021 30267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:26.174324 30267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:26.174479 30267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:26.181977 30267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:26.182202 30267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.66
I20240502 21:25:26.191474 30267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:32943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.29.82.66
--webserver_port=0
--tserver_master_addrs=127.29.82.126:40371
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.66
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:26.192927 30267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:26.194757 30267 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:26.207105 30275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:26.207623 30272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:26.208141 30273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:26.209022 30267 server_base.cc:1034] running on GCE node
I20240502 21:25:26.209573 30267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:26.210424 30267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:26.211684 30267 hybrid_clock.cc:648] HybridClock initialized: now 1714685126211620 us; error 37 us; skew 500 ppm
I20240502 21:25:26.211984 30267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:26.214160 30267 webserver.cc:460] Webserver started at http://127.29.82.66:44963/ using document root <none> and password file <none>
I20240502 21:25:26.214865 30267 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:26.214998 30267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:26.215304 30267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:26.217898 30267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "300b9807b6a34f93960edb9f67d22e59"
format_stamp: "Formatted at 2024-05-02 21:25:26 on dist-test-slave-xzsm"
I20240502 21:25:26.218647 30267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "300b9807b6a34f93960edb9f67d22e59"
format_stamp: "Formatted at 2024-05-02 21:25:26 on dist-test-slave-xzsm"
I20240502 21:25:26.223603 30267 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20240502 21:25:26.226856 30281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:26.228122 30267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:25:26.228278 30267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "300b9807b6a34f93960edb9f67d22e59"
format_stamp: "Formatted at 2024-05-02 21:25:26 on dist-test-slave-xzsm"
I20240502 21:25:26.228448 30267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:26.247375 30267 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:26.248211 30267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:26.248498 30267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:26.249428 30267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:26.250887 30267 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:26.250969 30267 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:26.251106 30267 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:26.251164 30267 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:26.287166 30267 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.66:46449
I20240502 21:25:26.287199 30398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.66:46449 every 8 connection(s)
I20240502 21:25:26.288465 30267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 21:25:26.295663 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 30267
I20240502 21:25:26.295871 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 21:25:26.299605 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.67:0
--local_ip_for_outbound_sockets=127.29.82.67
--webserver_interface=127.29.82.67
--webserver_port=0
--tserver_master_addrs=127.29.82.126:40371
--builtin_ntp_servers=127.29.82.84:32943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 21:25:26.301980 30399 heartbeater.cc:347] Connected to a master server at 127.29.82.126:40371
I20240502 21:25:26.302315 30399 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:26.303264 30399 heartbeater.cc:511] Master 127.29.82.126:40371 requested a full tablet report, sending...
I20240502 21:25:26.305047 30070 ts_manager.cc:194] Registered new tserver with Master: 300b9807b6a34f93960edb9f67d22e59 (127.29.82.66:46449)
I20240502 21:25:26.305886 30070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.66:50321
W20240502 21:25:26.634627 30402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:26.634910 30402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:26.635057 30402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:26.642705 30402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:26.642921 30402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.67
I20240502 21:25:26.651774 30402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:32943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.29.82.67
--webserver_port=0
--tserver_master_addrs=127.29.82.126:40371
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.67
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:26.653077 30402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:26.654839 30402 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:26.667443 30408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:26.668185 30410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:26.668902 30407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:26.669643 30402 server_base.cc:1034] running on GCE node
I20240502 21:25:26.670219 30402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:26.671073 30402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:26.672294 30402 hybrid_clock.cc:648] HybridClock initialized: now 1714685126672234 us; error 37 us; skew 500 ppm
I20240502 21:25:26.672587 30402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:26.674710 30402 webserver.cc:460] Webserver started at http://127.29.82.67:42885/ using document root <none> and password file <none>
I20240502 21:25:26.675428 30402 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:26.675560 30402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:26.675906 30402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:26.678486 30402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "18bfaec71ab2463b812930f7d9c1f7d5"
format_stamp: "Formatted at 2024-05-02 21:25:26 on dist-test-slave-xzsm"
I20240502 21:25:26.679245 30402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "18bfaec71ab2463b812930f7d9c1f7d5"
format_stamp: "Formatted at 2024-05-02 21:25:26 on dist-test-slave-xzsm"
I20240502 21:25:26.684357 30402 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20240502 21:25:26.687799 30416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:26.689050 30402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 21:25:26.689217 30402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "18bfaec71ab2463b812930f7d9c1f7d5"
format_stamp: "Formatted at 2024-05-02 21:25:26 on dist-test-slave-xzsm"
I20240502 21:25:26.689393 30402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:26.712867 30402 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:26.713744 30402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:26.714038 30402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:26.715025 30402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:26.716564 30402 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:26.716646 30402 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:26.716786 30402 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:26.716845 30402 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:26.749955 30402 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.67:38585
I20240502 21:25:26.749989 30532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.67:38585 every 8 connection(s)
I20240502 21:25:26.751276 30402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:25:26.758498 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 30402
I20240502 21:25:26.758728 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 21:25:26.762182 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.68:0
--local_ip_for_outbound_sockets=127.29.82.68
--webserver_interface=127.29.82.68
--webserver_port=0
--tserver_master_addrs=127.29.82.126:40371
--builtin_ntp_servers=127.29.82.84:32943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 21:25:26.764659 30533 heartbeater.cc:347] Connected to a master server at 127.29.82.126:40371
I20240502 21:25:26.764961 30533 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:26.765779 30533 heartbeater.cc:511] Master 127.29.82.126:40371 requested a full tablet report, sending...
I20240502 21:25:26.767694 30070 ts_manager.cc:194] Registered new tserver with Master: 18bfaec71ab2463b812930f7d9c1f7d5 (127.29.82.67:38585)
I20240502 21:25:26.768451 30070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.67:45677
I20240502 21:25:26.849611 30263 heartbeater.cc:503] Master 127.29.82.126:40371 was elected leader, sending a full tablet report...
W20240502 21:25:27.096658 30536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:27.097010 30536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:27.097167 30536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:27.104851 30536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:27.105055 30536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.68
I20240502 21:25:27.114187 30536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:32943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.29.82.68
--webserver_port=0
--tserver_master_addrs=127.29.82.126:40371
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.68
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:27.115620 30536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:27.117451 30536 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:27.129866 30542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:27.130435 30543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:27.130964 30545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:27.131886 30536 server_base.cc:1034] running on GCE node
I20240502 21:25:27.132472 30536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:27.133375 30536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:27.134586 30536 hybrid_clock.cc:648] HybridClock initialized: now 1714685127134525 us; error 32 us; skew 500 ppm
I20240502 21:25:27.134872 30536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:27.137077 30536 webserver.cc:460] Webserver started at http://127.29.82.68:45349/ using document root <none> and password file <none>
I20240502 21:25:27.137763 30536 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:27.137888 30536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:27.138183 30536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:27.140805 30536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "abb3a6b05d2e4d189f3518e773caf32f"
format_stamp: "Formatted at 2024-05-02 21:25:27 on dist-test-slave-xzsm"
I20240502 21:25:27.141598 30536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "abb3a6b05d2e4d189f3518e773caf32f"
format_stamp: "Formatted at 2024-05-02 21:25:27 on dist-test-slave-xzsm"
I20240502 21:25:27.146668 30536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 21:25:27.149958 30552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:27.151144 30536 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:27.151326 30536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "abb3a6b05d2e4d189f3518e773caf32f"
format_stamp: "Formatted at 2024-05-02 21:25:27 on dist-test-slave-xzsm"
I20240502 21:25:27.151485 30536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:27.171630 30536 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:27.172394 30536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:27.172686 30536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:27.173570 30536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:27.175011 30536 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:27.175086 30536 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:27.175212 30536 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:27.175269 30536 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:27.208711 30536 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.68:41809
I20240502 21:25:27.208762 30666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.68:41809 every 8 connection(s)
I20240502 21:25:27.210024 30536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:25:27.210685 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 30536
I20240502 21:25:27.210909 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 21:25:27.214654 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.69:0
--local_ip_for_outbound_sockets=127.29.82.69
--webserver_interface=127.29.82.69
--webserver_port=0
--tserver_master_addrs=127.29.82.126:40371
--builtin_ntp_servers=127.29.82.84:32943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 21:25:27.223601 30667 heartbeater.cc:347] Connected to a master server at 127.29.82.126:40371
I20240502 21:25:27.223913 30667 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:27.224892 30667 heartbeater.cc:511] Master 127.29.82.126:40371 requested a full tablet report, sending...
I20240502 21:25:27.226532 30070 ts_manager.cc:194] Registered new tserver with Master: abb3a6b05d2e4d189f3518e773caf32f (127.29.82.68:41809)
I20240502 21:25:27.227800 30070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.68:33717
I20240502 21:25:27.308451 30399 heartbeater.cc:503] Master 127.29.82.126:40371 was elected leader, sending a full tablet report...
W20240502 21:25:27.547626 30670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:27.547899 30670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:27.548034 30670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:27.555510 30670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:27.555828 30670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.69
I20240502 21:25:27.564898 30670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:32943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.29.82.69
--webserver_port=0
--tserver_master_addrs=127.29.82.126:40371
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.69
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:27.566210 30670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:27.567952 30670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:27.580000 30676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:27.580715 30677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:27.581431 30679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:27.582104 30670 server_base.cc:1034] running on GCE node
I20240502 21:25:27.582723 30670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:27.583648 30670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:27.584861 30670 hybrid_clock.cc:648] HybridClock initialized: now 1714685127584802 us; error 36 us; skew 500 ppm
I20240502 21:25:27.585150 30670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:27.587313 30670 webserver.cc:460] Webserver started at http://127.29.82.69:37721/ using document root <none> and password file <none>
I20240502 21:25:27.588040 30670 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:27.588172 30670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:27.588464 30670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:27.590909 30670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "761550e251f84fb1a17f23e741488c13"
format_stamp: "Formatted at 2024-05-02 21:25:27 on dist-test-slave-xzsm"
I20240502 21:25:27.591711 30670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "761550e251f84fb1a17f23e741488c13"
format_stamp: "Formatted at 2024-05-02 21:25:27 on dist-test-slave-xzsm"
I20240502 21:25:27.596740 30670 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 21:25:27.600003 30685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:27.601263 30670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 21:25:27.601461 30670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "761550e251f84fb1a17f23e741488c13"
format_stamp: "Formatted at 2024-05-02 21:25:27 on dist-test-slave-xzsm"
I20240502 21:25:27.601653 30670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:27.626734 30670 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:27.627535 30670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:27.627875 30670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:27.628830 30670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:27.630350 30670 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:27.630431 30670 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:27.630566 30670 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:27.630625 30670 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:27.661260 30670 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.69:41853
I20240502 21:25:27.661311 30799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.69:41853 every 8 connection(s)
I20240502 21:25:27.662557 30670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:25:27.663935 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 30670
I20240502 21:25:27.664142 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 21:25:27.675107 30800 heartbeater.cc:347] Connected to a master server at 127.29.82.126:40371
I20240502 21:25:27.675354 30800 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:27.676154 30800 heartbeater.cc:511] Master 127.29.82.126:40371 requested a full tablet report, sending...
I20240502 21:25:27.677414 30070 ts_manager.cc:194] Registered new tserver with Master: 761550e251f84fb1a17f23e741488c13 (127.29.82.69:41853)
I20240502 21:25:27.678145 30070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.69:46395
I20240502 21:25:27.681030 30025 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 21:25:27.701943 30070 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40596:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 21:25:27.747200 30466 tablet_service.cc:1467] Processing CreateTablet for tablet 4750dd8c71c6466191251d7b440f5c5d (DEFAULT_TABLE table=TestTable [id=b71348dbec5c45388c64127ee83933ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:27.747216 30196 tablet_service.cc:1467] Processing CreateTablet for tablet 4750dd8c71c6466191251d7b440f5c5d (DEFAULT_TABLE table=TestTable [id=b71348dbec5c45388c64127ee83933ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:27.747213 30330 tablet_service.cc:1467] Processing CreateTablet for tablet 4750dd8c71c6466191251d7b440f5c5d (DEFAULT_TABLE table=TestTable [id=b71348dbec5c45388c64127ee83933ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:27.748687 30466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4750dd8c71c6466191251d7b440f5c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:27.748773 30196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4750dd8c71c6466191251d7b440f5c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:27.749156 30330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4750dd8c71c6466191251d7b440f5c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:27.762948 30820 tablet_bootstrap.cc:492] T 4750dd8c71c6466191251d7b440f5c5d P 18bfaec71ab2463b812930f7d9c1f7d5: Bootstrap starting.
I20240502 21:25:27.764178 30819 tablet_bootstrap.cc:492] T 4750dd8c71c6466191251d7b440f5c5d P 6402e732f88a4bbea512b429a7e2e6ad: Bootstrap starting.
I20240502 21:25:27.765297 30821 tablet_bootstrap.cc:492] T 4750dd8c71c6466191251d7b440f5c5d P 300b9807b6a34f93960edb9f67d22e59: Bootstrap starting.
I20240502 21:25:27.768357 30820 tablet_bootstrap.cc:654] T 4750dd8c71c6466191251d7b440f5c5d P 18bfaec71ab2463b812930f7d9c1f7d5: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:27.768357 30819 tablet_bootstrap.cc:654] T 4750dd8c71c6466191251d7b440f5c5d P 6402e732f88a4bbea512b429a7e2e6ad: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:27.768970 30821 tablet_bootstrap.cc:654] T 4750dd8c71c6466191251d7b440f5c5d P 300b9807b6a34f93960edb9f67d22e59: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:27.769719 30819 log.cc:824] T 4750dd8c71c6466191251d7b440f5c5d P 6402e732f88a4bbea512b429a7e2e6ad: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:27.769723 30820 log.cc:824] T 4750dd8c71c6466191251d7b440f5c5d P 18bfaec71ab2463b812930f7d9c1f7d5: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:27.769986 30821 log.cc:824] T 4750dd8c71c6466191251d7b440f5c5d P 300b9807b6a34f93960edb9f67d22e59: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:27.770740 30533 heartbeater.cc:503] Master 127.29.82.126:40371 was elected leader, sending a full tablet report...
I20240502 21:25:27.773226 30819 tablet_bootstrap.cc:492] T 4750dd8c71c6466191251d7b440f5c5d P 6402e732f88a4bbea512b429a7e2e6ad: No bootstrap required, opened a new log
I20240502 21:25:27.773582 30819 ts_tablet_manager.cc:1367] T 4750dd8c71c6466191251d7b440f5c5d P 6402e732f88a4bbea512b429a7e2e6ad: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20240502 21:25:27.773808 30820 tablet_bootstrap.cc:492] T 4750dd8c71c6466191251d7b440f5c5d P 18bfaec71ab2463b812930f7d9c1f7d5: No bootstrap required, opened a new log
I20240502 21:25:27.774147 30820 ts_tablet_manager.cc:1367] T 4750dd8c71c6466191251d7b440f5c5d P 18bfaec71ab2463b812930f7d9c1f7d5: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20240502 21:25:27.779358 30821 tablet_bootstrap.cc:492] T 4750dd8c71c6466191251d7b440f5c5d P 300b9807b6a34f93960edb9f67d22e59: No bootstrap required, opened a new log
I20240502 21:25:27.779693 30821 ts_tablet_manager.cc:1367] T 4750dd8c71c6466191251d7b440f5c5d P 300b9807b6a34f93960edb9f67d22e59: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20240502 21:25:27.782775 30819 raft_consensus.cc:348] T 4750dd8c71c6466191251d7b440f5c5d P 6402e732f88a4bbea512b429a7e2e6ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6402e732f88a4bbea512b429a7e2e6ad" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 39605 } } peers { permanent_uuid: "300b9807b6a34f93960edb9f67d22e59" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 46449 } } peers { permanent_uuid: "18bfaec71ab2463b812930f7d9c1f7d5" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 38585 } }
I20240502 21:25:27.783239 30819 raft_consensus.cc:374] T 4750dd8c71c6466191251d7b440f5c5d P 6402e732f88a4bbea512b429a7e2e6ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:27.783416 30819 raft_consensus.cc:729] T 4750dd8c71c6466191251d7b440f5c5d P 6402e732f88a4bbea512b429a7e2e6ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6402e732f88a4bbea512b429a7e2e6ad, State: Initialized, Role: FOLLOWER
I20240502 21:25:27.783330 30820 raft_consensus.cc:348] T 4750dd8c71c6466191251d7b440f5c5d P 18bfaec71ab2463b812930f7d9c1f7d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6402e732f88a4bbea512b429a7e2e6ad" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 39605 } } peers { permanent_uuid: "300b9807b6a34f93960edb9f67d22e59" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 46449 } } peers { permanent_uuid: "18bfaec71ab2463b812930f7d9c1f7d5" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 38585 } }
I20240502 21:25:27.783758 30820 raft_consensus.cc:374] T 4750dd8c71c6466191251d7b440f5c5d P 18bfaec71ab2463b812930f7d9c1f7d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:27.783888 30820 raft_consensus.cc:729] T 4750dd8c71c6466191251d7b440f5c5d P 18bfaec71ab2463b812930f7d9c1f7d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18bfaec71ab2463b812930f7d9c1f7d5, State: Initialized, Role: FOLLOWER
I20240502 21:25:27.784320 30819 consensus_queue.cc:260] T 4750dd8c71c6466191251d7b440f5c5d P 6402e732f88a4bbea512b429a7e2e6ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6402e732f88a4bbea512b429a7e2e6ad" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 39605 } } peers { permanent_uuid: "300b9807b6a34f93960edb9f67d22e59" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 46449 } } peers { permanent_uuid: "18bfaec71ab2463b812930f7d9c1f7d5" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 38585 } }
I20240502 21:25:27.784528 30820 consensus_queue.cc:260] T 4750dd8c71c6466191251d7b440f5c5d P 18bfaec71ab2463b812930f7d9c1f7d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6402e732f88a4bbea512b429a7e2e6ad" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 39605 } } peers { permanent_uuid: "300b9807b6a34f93960edb9f67d22e59" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 46449 } } peers { permanent_uuid: "18bfaec71ab2463b812930f7d9c1f7d5" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 38585 } }
I20240502 21:25:27.786756 30819 ts_tablet_manager.cc:1398] T 4750dd8c71c6466191251d7b440f5c5d P 6402e732f88a4bbea512b429a7e2e6ad: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20240502 21:25:27.786664 30821 raft_consensus.cc:348] T 4750dd8c71c6466191251d7b440f5c5d P 300b9807b6a34f93960edb9f67d22e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6402e732f88a4bbea512b429a7e2e6ad" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 39605 } } peers { permanent_uuid: "300b9807b6a34f93960edb9f67d22e59" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 46449 } } peers { permanent_uuid: "18bfaec71ab2463b812930f7d9c1f7d5" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 38585 } }
I20240502 21:25:27.787122 30821 raft_consensus.cc:374] T 4750dd8c71c6466191251d7b440f5c5d P 300b9807b6a34f93960edb9f67d22e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:27.787267 30821 raft_consensus.cc:729] T 4750dd8c71c6466191251d7b440f5c5d P 300b9807b6a34f93960edb9f67d22e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 300b9807b6a34f93960edb9f67d22e59, State: Initialized, Role: FOLLOWER
I20240502 21:25:27.788043 30821 consensus_queue.cc:260] T 4750dd8c71c6466191251d7b440f5c5d P 300b9807b6a34f93960edb9f67d22e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6402e732f88a4bbea512b429a7e2e6ad" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 39605 } } peers { permanent_uuid: "300b9807b6a34f93960edb9f67d22e59" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 46449 } } peers { permanent_uuid: "18bfaec71ab2463b812930f7d9c1f7d5" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 38585 } }
I20240502 21:25:27.788575 30820 ts_tablet_manager.cc:1398] T 4750dd8c71c6466191251d7b440f5c5d P 18bfaec71ab2463b812930f7d9c1f7d5: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20240502 21:25:27.790213 30821 ts_tablet_manager.cc:1398] T 4750dd8c71c6466191251d7b440f5c5d P 300b9807b6a34f93960edb9f67d22e59: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
W20240502 21:25:27.790894 30400 tablet.cc:2369] T 4750dd8c71c6466191251d7b440f5c5d P 300b9807b6a34f93960edb9f67d22e59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:27.823149 30825 raft_consensus.cc:482] T 4750dd8c71c6466191251d7b440f5c5d P 6402e732f88a4bbea512b429a7e2e6ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:27.823413 30825 raft_consensus.cc:504] T 4750dd8c71c6466191251d7b440f5c5d P 6402e732f88a4bbea512b429a7e2e6ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6402e732f88a4bbea512b429a7e2e6ad" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 39605 } } peers { permanent_uuid: "300b9807b6a34f93960edb9f67d22e59" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 46449 } } peers { permanent_uuid: "18bfaec71ab2463b812930f7d9c1f7d5" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 38585 } }
I20240502 21:25:27.825173 30825 leader_election.cc:288] T 4750dd8c71c6466191251d7b440f5c5d P 6402e732f88a4bbea512b429a7e2e6ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 300b9807b6a34f93960edb9f67d22e59 (127.29.82.66:46449), 18bfaec71ab2463b812930f7d9c1f7d5 (127.29.82.67:38585)
W20240502 21:25:27.830431 30264 tablet.cc:2369] T 4750dd8c71c6466191251d7b440f5c5d P 6402e732f88a4bbea512b429a7e2e6ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:27.834221 30350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4750dd8c71c6466191251d7b440f5c5d" candidate_uuid: "6402e732f88a4bbea512b429a7e2e6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "300b9807b6a34f93960edb9f67d22e59" is_pre_election: true
I20240502 21:25:27.834335 30487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4750dd8c71c6466191251d7b440f5c5d" candidate_uuid: "6402e732f88a4bbea512b429a7e2e6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18bfaec71ab2463b812930f7d9c1f7d5" is_pre_election: true
I20240502 21:25:27.834719 30350 raft_consensus.cc:2454] T 4750dd8c71c6466191251d7b440f5c5d P 300b9807b6a34f93960edb9f67d22e59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6402e732f88a4bbea512b429a7e2e6ad in term 0.
I20240502 21:25:27.834810 30487 raft_consensus.cc:2454] T 4750dd8c71c6466191251d7b440f5c5d P 18bfaec71ab2463b812930f7d9c1f7d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6402e732f88a4bbea512b429a7e2e6ad in term 0.
I20240502 21:25:27.835395 30148 leader_election.cc:302] T 4750dd8c71c6466191251d7b440f5c5d P 6402e732f88a4bbea512b429a7e2e6ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 300b9807b6a34f93960edb9f67d22e59, 6402e732f88a4bbea512b429a7e2e6ad; no voters:
I20240502 21:25:27.835762 30825 raft_consensus.cc:2781] T 4750dd8c71c6466191251d7b440f5c5d P 6402e732f88a4bbea512b429a7e2e6ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:27.835922 30825 raft_consensus.cc:482] T 4750dd8c71c6466191251d7b440f5c5d P 6402e732f88a4bbea512b429a7e2e6ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:27.836063 30825 raft_consensus.cc:3037] T 4750dd8c71c6466191251d7b440f5c5d P 6402e732f88a4bbea512b429a7e2e6ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:27.838626 30825 raft_consensus.cc:504] T 4750dd8c71c6466191251d7b440f5c5d P 6402e732f88a4bbea512b429a7e2e6ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6402e732f88a4bbea512b429a7e2e6ad" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 39605 } } peers { permanent_uuid: "300b9807b6a34f93960edb9f67d22e59" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 46449 } } peers { permanent_uuid: "18bfaec71ab2463b812930f7d9c1f7d5" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 38585 } }
I20240502 21:25:27.839285 30825 leader_election.cc:288] T 4750dd8c71c6466191251d7b440f5c5d P 6402e732f88a4bbea512b429a7e2e6ad [CANDIDATE]: Term 1 election: Requested vote from peers 300b9807b6a34f93960edb9f67d22e59 (127.29.82.66:46449), 18bfaec71ab2463b812930f7d9c1f7d5 (127.29.82.67:38585)
I20240502 21:25:27.839774 30487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4750dd8c71c6466191251d7b440f5c5d" candidate_uuid: "6402e732f88a4bbea512b429a7e2e6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18bfaec71ab2463b812930f7d9c1f7d5"
I20240502 21:25:27.839773 30350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4750dd8c71c6466191251d7b440f5c5d" candidate_uuid: "6402e732f88a4bbea512b429a7e2e6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "300b9807b6a34f93960edb9f67d22e59"
I20240502 21:25:27.839969 30487 raft_consensus.cc:3037] T 4750dd8c71c6466191251d7b440f5c5d P 18bfaec71ab2463b812930f7d9c1f7d5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:27.839970 30350 raft_consensus.cc:3037] T 4750dd8c71c6466191251d7b440f5c5d P 300b9807b6a34f93960edb9f67d22e59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:27.843075 30350 raft_consensus.cc:2454] T 4750dd8c71c6466191251d7b440f5c5d P 300b9807b6a34f93960edb9f67d22e59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6402e732f88a4bbea512b429a7e2e6ad in term 1.
I20240502 21:25:27.843075 30487 raft_consensus.cc:2454] T 4750dd8c71c6466191251d7b440f5c5d P 18bfaec71ab2463b812930f7d9c1f7d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6402e732f88a4bbea512b429a7e2e6ad in term 1.
I20240502 21:25:27.843444 30148 leader_election.cc:302] T 4750dd8c71c6466191251d7b440f5c5d P 6402e732f88a4bbea512b429a7e2e6ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 300b9807b6a34f93960edb9f67d22e59, 6402e732f88a4bbea512b429a7e2e6ad; no voters:
I20240502 21:25:27.843703 30825 raft_consensus.cc:2781] T 4750dd8c71c6466191251d7b440f5c5d P 6402e732f88a4bbea512b429a7e2e6ad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:27.844156 30825 raft_consensus.cc:686] T 4750dd8c71c6466191251d7b440f5c5d P 6402e732f88a4bbea512b429a7e2e6ad [term 1 LEADER]: Becoming Leader. State: Replica: 6402e732f88a4bbea512b429a7e2e6ad, State: Running, Role: LEADER
I20240502 21:25:27.844604 30825 consensus_queue.cc:237] T 4750dd8c71c6466191251d7b440f5c5d P 6402e732f88a4bbea512b429a7e2e6ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6402e732f88a4bbea512b429a7e2e6ad" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 39605 } } peers { permanent_uuid: "300b9807b6a34f93960edb9f67d22e59" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 46449 } } peers { permanent_uuid: "18bfaec71ab2463b812930f7d9c1f7d5" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 38585 } }
I20240502 21:25:27.848829 30068 catalog_manager.cc:5484] T 4750dd8c71c6466191251d7b440f5c5d P 6402e732f88a4bbea512b429a7e2e6ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 6402e732f88a4bbea512b429a7e2e6ad (127.29.82.65). New cstate: current_term: 1 leader_uuid: "6402e732f88a4bbea512b429a7e2e6ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6402e732f88a4bbea512b429a7e2e6ad" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 39605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "300b9807b6a34f93960edb9f67d22e59" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 46449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18bfaec71ab2463b812930f7d9c1f7d5" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 38585 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:27.870604 30025 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 21:25:27.873152 30025 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6402e732f88a4bbea512b429a7e2e6ad to finish bootstrapping
I20240502 21:25:27.882592 30025 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 300b9807b6a34f93960edb9f67d22e59 to finish bootstrapping
I20240502 21:25:27.890187 30025 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 18bfaec71ab2463b812930f7d9c1f7d5 to finish bootstrapping
I20240502 21:25:27.897588 30025 ts_itest-base.cc:233] Waiting for 0 tablets on tserver abb3a6b05d2e4d189f3518e773caf32f to finish bootstrapping
I20240502 21:25:27.906086 30025 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 761550e251f84fb1a17f23e741488c13 to finish bootstrapping
W20240502 21:25:28.003837 30534 tablet.cc:2369] T 4750dd8c71c6466191251d7b440f5c5d P 18bfaec71ab2463b812930f7d9c1f7d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:28.230878 30667 heartbeater.cc:503] Master 127.29.82.126:40371 was elected leader, sending a full tablet report...
I20240502 21:25:28.248569 30825 consensus_queue.cc:1035] T 4750dd8c71c6466191251d7b440f5c5d P 6402e732f88a4bbea512b429a7e2e6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "300b9807b6a34f93960edb9f67d22e59" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 46449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:28.256840 30825 consensus_queue.cc:1035] T 4750dd8c71c6466191251d7b440f5c5d P 6402e732f88a4bbea512b429a7e2e6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "18bfaec71ab2463b812930f7d9c1f7d5" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:28.388871 30330 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:28.391037 30466 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:28.393061 30196 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:28.397434 30734 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:28.409363 30600 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:28.578313 30466 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:28.590417 30734 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:28.591184 30330 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:28.599870 30196 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:28.600777 30600 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:28.680387 30800 heartbeater.cc:503] Master 127.29.82.126:40371 was elected leader, sending a full tablet report...
I20240502 21:25:28.755633 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 30132
I20240502 21:25:28.757464 30257 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:28.853101 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 30132
I20240502 21:25:28.880089 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 30267
I20240502 21:25:28.881249 30393 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:28.972237 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 30267
I20240502 21:25:28.992336 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 30402
I20240502 21:25:28.993374 30528 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:29.085287 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 30402
I20240502 21:25:29.104707 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 30536
I20240502 21:25:29.105813 30661 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:29.190415 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 30536
I20240502 21:25:29.208695 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 30670
I20240502 21:25:29.209775 30794 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:29.294375 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 30670
I20240502 21:25:29.313290 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 30038
I20240502 21:25:29.314514 30103 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:29.427105 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 30038
2024-05-02T21:25:29Z chronyd exiting
[ OK ] AdminCliTest.RebalancerReportOnly (4737 ms)
[----------] 1 test from AdminCliTest (4737 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/0
I20240502 21:25:29.473618 30025 test_util.cc:268] Using random seed: 1040924199
I20240502 21:25:29.475510 30025 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:25:29.475634 30025 ts_itest-base.cc:113] --------------
I20240502 21:25:29.475703 30025 ts_itest-base.cc:114] 5 tablet servers
I20240502 21:25:29.475778 30025 ts_itest-base.cc:115] 3 replicas per TS
I20240502 21:25:29.475821 30025 ts_itest-base.cc:116] --------------
2024-05-02T21:25:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:25:29Z Disabled control of system clock
I20240502 21:25:29.501456 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.82.126:41665
--webserver_interface=127.29.82.126
--webserver_port=0
--builtin_ntp_servers=127.29.82.84:38471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.82.126:41665
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240502 21:25:29.832415 30925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:29.832726 30925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:29.832870 30925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:29.840202 30925 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:25:29.840322 30925 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 21:25:29.840397 30925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:29.840477 30925 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 21:25:29.840545 30925 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 21:25:29.849436 30925 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:38471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.29.82.126:41665
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.82.126:41665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.29.82.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:29.850744 30925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:29.852488 30925 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:29.863366 30933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:29.863951 30931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:29.864698 30930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:29.865450 30925 server_base.cc:1034] running on GCE node
I20240502 21:25:29.866171 30925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:29.867522 30925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:29.868767 30925 hybrid_clock.cc:648] HybridClock initialized: now 1714685129868719 us; error 43 us; skew 500 ppm
I20240502 21:25:29.869093 30925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:29.871412 30925 webserver.cc:460] Webserver started at http://127.29.82.126:33447/ using document root <none> and password file <none>
I20240502 21:25:29.872164 30925 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:29.872284 30925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:29.872613 30925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:29.876966 30925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "177c19f8675246bcb3a7ad949dea04db"
format_stamp: "Formatted at 2024-05-02 21:25:29 on dist-test-slave-xzsm"
I20240502 21:25:29.877813 30925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "177c19f8675246bcb3a7ad949dea04db"
format_stamp: "Formatted at 2024-05-02 21:25:29 on dist-test-slave-xzsm"
I20240502 21:25:29.883188 30925 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20240502 21:25:29.886598 30939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:29.888031 30925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:25:29.888206 30925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal
uuid: "177c19f8675246bcb3a7ad949dea04db"
format_stamp: "Formatted at 2024-05-02 21:25:29 on dist-test-slave-xzsm"
I20240502 21:25:29.888410 30925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:29.901824 30925 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:29.902658 30925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:29.902954 30925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:29.921523 30925 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.126:41665
I20240502 21:25:29.921532 30993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.126:41665 every 8 connection(s)
I20240502 21:25:29.922923 30925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 21:25:29.926627 30994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:29.928931 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 30925
I20240502 21:25:29.929180 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 21:25:29.937662 30994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 177c19f8675246bcb3a7ad949dea04db: Bootstrap starting.
I20240502 21:25:29.940779 30994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 177c19f8675246bcb3a7ad949dea04db: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:29.942040 30994 log.cc:824] T 00000000000000000000000000000000 P 177c19f8675246bcb3a7ad949dea04db: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:29.944875 30994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 177c19f8675246bcb3a7ad949dea04db: No bootstrap required, opened a new log
I20240502 21:25:29.950954 30994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 177c19f8675246bcb3a7ad949dea04db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177c19f8675246bcb3a7ad949dea04db" member_type: VOTER last_known_addr { host: "127.29.82.126" port: 41665 } }
I20240502 21:25:29.951354 30994 raft_consensus.cc:374] T 00000000000000000000000000000000 P 177c19f8675246bcb3a7ad949dea04db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:29.951468 30994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 177c19f8675246bcb3a7ad949dea04db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 177c19f8675246bcb3a7ad949dea04db, State: Initialized, Role: FOLLOWER
I20240502 21:25:29.952090 30994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 177c19f8675246bcb3a7ad949dea04db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177c19f8675246bcb3a7ad949dea04db" member_type: VOTER last_known_addr { host: "127.29.82.126" port: 41665 } }
I20240502 21:25:29.952293 30994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 177c19f8675246bcb3a7ad949dea04db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:29.952416 30994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 177c19f8675246bcb3a7ad949dea04db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:29.952558 30994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 177c19f8675246bcb3a7ad949dea04db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:29.954526 30994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 177c19f8675246bcb3a7ad949dea04db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177c19f8675246bcb3a7ad949dea04db" member_type: VOTER last_known_addr { host: "127.29.82.126" port: 41665 } }
I20240502 21:25:29.954988 30994 leader_election.cc:302] T 00000000000000000000000000000000 P 177c19f8675246bcb3a7ad949dea04db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 177c19f8675246bcb3a7ad949dea04db; no voters:
I20240502 21:25:29.955644 30994 leader_election.cc:288] T 00000000000000000000000000000000 P 177c19f8675246bcb3a7ad949dea04db [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:29.955782 30999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 177c19f8675246bcb3a7ad949dea04db [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:29.956437 30999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 177c19f8675246bcb3a7ad949dea04db [term 1 LEADER]: Becoming Leader. State: Replica: 177c19f8675246bcb3a7ad949dea04db, State: Running, Role: LEADER
I20240502 21:25:29.956982 30999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 177c19f8675246bcb3a7ad949dea04db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177c19f8675246bcb3a7ad949dea04db" member_type: VOTER last_known_addr { host: "127.29.82.126" port: 41665 } }
I20240502 21:25:29.957552 30994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 177c19f8675246bcb3a7ad949dea04db [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:25:29.959892 31000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 177c19f8675246bcb3a7ad949dea04db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "177c19f8675246bcb3a7ad949dea04db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177c19f8675246bcb3a7ad949dea04db" member_type: VOTER last_known_addr { host: "127.29.82.126" port: 41665 } } }
I20240502 21:25:29.960299 31000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 177c19f8675246bcb3a7ad949dea04db [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:29.960541 31001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 177c19f8675246bcb3a7ad949dea04db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 177c19f8675246bcb3a7ad949dea04db. Latest consensus state: current_term: 1 leader_uuid: "177c19f8675246bcb3a7ad949dea04db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177c19f8675246bcb3a7ad949dea04db" member_type: VOTER last_known_addr { host: "127.29.82.126" port: 41665 } } }
I20240502 21:25:29.962244 31001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 177c19f8675246bcb3a7ad949dea04db [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:29.963862 31007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:25:29.969002 31007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:25:29.977699 31007 catalog_manager.cc:1347] Generated new cluster ID: 38a5d4d4a7b64c0e956863d74d2f1ddc
I20240502 21:25:29.977810 31007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:25:29.999166 31007 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:25:30.000463 31007 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:25:30.014031 31007 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 177c19f8675246bcb3a7ad949dea04db: Generated new TSK 0
I20240502 21:25:30.014976 31007 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:25:30.029811 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.65:0
--local_ip_for_outbound_sockets=127.29.82.65
--webserver_interface=127.29.82.65
--webserver_port=0
--tserver_master_addrs=127.29.82.126:41665
--builtin_ntp_servers=127.29.82.84:38471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 21:25:30.359464 31018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:30.359781 31018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:30.359926 31018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:30.367352 31018 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:25:30.367465 31018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:30.367660 31018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.65
I20240502 21:25:30.376591 31018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:38471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.29.82.65
--webserver_port=0
--tserver_master_addrs=127.29.82.126:41665
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.65
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:30.377960 31018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:30.379797 31018 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:30.393324 31024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:30.393952 31027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:30.394429 31023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:30.395293 31018 server_base.cc:1034] running on GCE node
I20240502 21:25:30.396155 31018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:30.397500 31018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:30.398753 31018 hybrid_clock.cc:648] HybridClock initialized: now 1714685130398681 us; error 55 us; skew 500 ppm
I20240502 21:25:30.399155 31018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:30.406859 31018 webserver.cc:460] Webserver started at http://127.29.82.65:44015/ using document root <none> and password file <none>
I20240502 21:25:30.407629 31018 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:30.407737 31018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:30.408033 31018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:30.410637 31018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "9d3ce5eec771415a89fd7390c6ddaa7a"
format_stamp: "Formatted at 2024-05-02 21:25:30 on dist-test-slave-xzsm"
I20240502 21:25:30.411365 31018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "9d3ce5eec771415a89fd7390c6ddaa7a"
format_stamp: "Formatted at 2024-05-02 21:25:30 on dist-test-slave-xzsm"
I20240502 21:25:30.416509 31018 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20240502 21:25:30.419765 31034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:30.421231 31018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 21:25:30.421406 31018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "9d3ce5eec771415a89fd7390c6ddaa7a"
format_stamp: "Formatted at 2024-05-02 21:25:30 on dist-test-slave-xzsm"
I20240502 21:25:30.421608 31018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:30.434932 31018 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:30.435803 31018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:30.436060 31018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:30.436940 31018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:30.438357 31018 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:30.438447 31018 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:30.438562 31018 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:30.438647 31018 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:30.472626 31018 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.65:44193
I20240502 21:25:30.472707 31150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.65:44193 every 8 connection(s)
I20240502 21:25:30.474032 31018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 21:25:30.478142 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 31018
I20240502 21:25:30.478363 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 21:25:30.482378 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.66:0
--local_ip_for_outbound_sockets=127.29.82.66
--webserver_interface=127.29.82.66
--webserver_port=0
--tserver_master_addrs=127.29.82.126:41665
--builtin_ntp_servers=127.29.82.84:38471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:25:30.488687 31151 heartbeater.cc:347] Connected to a master server at 127.29.82.126:41665
I20240502 21:25:30.489006 31151 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:30.489795 31151 heartbeater.cc:511] Master 127.29.82.126:41665 requested a full tablet report, sending...
I20240502 21:25:30.492151 30957 ts_manager.cc:194] Registered new tserver with Master: 9d3ce5eec771415a89fd7390c6ddaa7a (127.29.82.65:44193)
I20240502 21:25:30.494395 30957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.65:48427
W20240502 21:25:30.808712 31156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:30.808981 31156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:30.809078 31156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:30.816556 31156 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:25:30.816684 31156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:30.816809 31156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.66
I20240502 21:25:30.825922 31156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:38471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.29.82.66
--webserver_port=0
--tserver_master_addrs=127.29.82.126:41665
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.66
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:30.827304 31156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:30.829049 31156 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:30.840987 31163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:30.841609 31162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:30.842298 31165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:30.842944 31156 server_base.cc:1034] running on GCE node
I20240502 21:25:30.843585 31156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:30.844491 31156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:30.845705 31156 hybrid_clock.cc:648] HybridClock initialized: now 1714685130845670 us; error 22 us; skew 500 ppm
I20240502 21:25:30.845999 31156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:30.848102 31156 webserver.cc:460] Webserver started at http://127.29.82.66:35081/ using document root <none> and password file <none>
I20240502 21:25:30.848780 31156 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:30.848896 31156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:30.849205 31156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:30.851735 31156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "4242a19eef2f48b8a11e0f2e8702a3a9"
format_stamp: "Formatted at 2024-05-02 21:25:30 on dist-test-slave-xzsm"
I20240502 21:25:30.852476 31156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "4242a19eef2f48b8a11e0f2e8702a3a9"
format_stamp: "Formatted at 2024-05-02 21:25:30 on dist-test-slave-xzsm"
I20240502 21:25:30.857198 31156 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 21:25:30.860570 31171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:30.861793 31156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 21:25:30.861938 31156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "4242a19eef2f48b8a11e0f2e8702a3a9"
format_stamp: "Formatted at 2024-05-02 21:25:30 on dist-test-slave-xzsm"
I20240502 21:25:30.862116 31156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:30.892385 31156 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:30.893375 31156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:30.893687 31156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:30.894709 31156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:30.896253 31156 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:30.896354 31156 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:30.896479 31156 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:30.896551 31156 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:30.928442 31156 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.66:39693
I20240502 21:25:30.928561 31288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.66:39693 every 8 connection(s)
I20240502 21:25:30.929805 31156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 21:25:30.933413 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 31156
I20240502 21:25:30.933609 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 21:25:30.937252 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.67:0
--local_ip_for_outbound_sockets=127.29.82.67
--webserver_interface=127.29.82.67
--webserver_port=0
--tserver_master_addrs=127.29.82.126:41665
--builtin_ntp_servers=127.29.82.84:38471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:25:30.944409 31289 heartbeater.cc:347] Connected to a master server at 127.29.82.126:41665
I20240502 21:25:30.944743 31289 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:30.945616 31289 heartbeater.cc:511] Master 127.29.82.126:41665 requested a full tablet report, sending...
I20240502 21:25:30.947171 30957 ts_manager.cc:194] Registered new tserver with Master: 4242a19eef2f48b8a11e0f2e8702a3a9 (127.29.82.66:39693)
I20240502 21:25:30.947973 30957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.66:39173
W20240502 21:25:31.261775 31292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:31.262042 31292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:31.262187 31292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:31.269541 31292 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:25:31.269680 31292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:31.269835 31292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.67
I20240502 21:25:31.278663 31292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:38471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.29.82.67
--webserver_port=0
--tserver_master_addrs=127.29.82.126:41665
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.67
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:31.279999 31292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:31.281764 31292 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:31.294164 31297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:31.294894 31300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:31.295482 31298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:31.296234 31292 server_base.cc:1034] running on GCE node
I20240502 21:25:31.296792 31292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:31.297683 31292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:31.298882 31292 hybrid_clock.cc:648] HybridClock initialized: now 1714685131298826 us; error 53 us; skew 500 ppm
I20240502 21:25:31.299203 31292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:31.301451 31292 webserver.cc:460] Webserver started at http://127.29.82.67:40001/ using document root <none> and password file <none>
I20240502 21:25:31.302181 31292 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:31.302295 31292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:31.302629 31292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:31.305202 31292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "2a3cfd9dd6f844e8a57693f00d34c41a"
format_stamp: "Formatted at 2024-05-02 21:25:31 on dist-test-slave-xzsm"
I20240502 21:25:31.306035 31292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "2a3cfd9dd6f844e8a57693f00d34c41a"
format_stamp: "Formatted at 2024-05-02 21:25:31 on dist-test-slave-xzsm"
I20240502 21:25:31.311069 31292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 21:25:31.314429 31306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:31.315775 31292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:25:31.315954 31292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "2a3cfd9dd6f844e8a57693f00d34c41a"
format_stamp: "Formatted at 2024-05-02 21:25:31 on dist-test-slave-xzsm"
I20240502 21:25:31.316140 31292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:31.347268 31292 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:31.348196 31292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:31.348440 31292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:31.349359 31292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:31.350831 31292 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:31.350916 31292 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:31.351006 31292 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:31.351061 31292 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:31.385499 31292 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.67:37759
I20240502 21:25:31.385562 31424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.67:37759 every 8 connection(s)
I20240502 21:25:31.386822 31292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:25:31.387218 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 31292
I20240502 21:25:31.387380 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 21:25:31.391616 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.68:0
--local_ip_for_outbound_sockets=127.29.82.68
--webserver_interface=127.29.82.68
--webserver_port=0
--tserver_master_addrs=127.29.82.126:41665
--builtin_ntp_servers=127.29.82.84:38471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:25:31.400200 31425 heartbeater.cc:347] Connected to a master server at 127.29.82.126:41665
I20240502 21:25:31.400483 31425 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:31.401194 31425 heartbeater.cc:511] Master 127.29.82.126:41665 requested a full tablet report, sending...
I20240502 21:25:31.402582 30957 ts_manager.cc:194] Registered new tserver with Master: 2a3cfd9dd6f844e8a57693f00d34c41a (127.29.82.67:37759)
I20240502 21:25:31.403371 30957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.67:56819
I20240502 21:25:31.497354 31151 heartbeater.cc:503] Master 127.29.82.126:41665 was elected leader, sending a full tablet report...
W20240502 21:25:31.725133 31428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:31.725430 31428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:31.725572 31428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:31.732991 31428 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:25:31.733135 31428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:31.733296 31428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.68
I20240502 21:25:31.742027 31428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:38471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.29.82.68
--webserver_port=0
--tserver_master_addrs=127.29.82.126:41665
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.68
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:31.743332 31428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:31.745093 31428 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:31.757021 31433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:31.757730 31436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:31.758428 31434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:31.759068 31428 server_base.cc:1034] running on GCE node
I20240502 21:25:31.759670 31428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:31.760799 31428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:31.761991 31428 hybrid_clock.cc:648] HybridClock initialized: now 1714685131761942 us; error 45 us; skew 500 ppm
I20240502 21:25:31.762280 31428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:31.764561 31428 webserver.cc:460] Webserver started at http://127.29.82.68:45871/ using document root <none> and password file <none>
I20240502 21:25:31.765249 31428 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:31.765375 31428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:31.765712 31428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:31.768307 31428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "09ad321209604f9eb2c06cbfcbb8bfdf"
format_stamp: "Formatted at 2024-05-02 21:25:31 on dist-test-slave-xzsm"
I20240502 21:25:31.769052 31428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "09ad321209604f9eb2c06cbfcbb8bfdf"
format_stamp: "Formatted at 2024-05-02 21:25:31 on dist-test-slave-xzsm"
I20240502 21:25:31.774077 31428 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 21:25:31.777271 31443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:31.778479 31428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 21:25:31.778625 31428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "09ad321209604f9eb2c06cbfcbb8bfdf"
format_stamp: "Formatted at 2024-05-02 21:25:31 on dist-test-slave-xzsm"
I20240502 21:25:31.778805 31428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:31.808036 31428 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:31.808948 31428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:31.809219 31428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:31.810212 31428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:31.812004 31428 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:31.812134 31428 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:31.812284 31428 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:31.812361 31428 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:31.853348 31428 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.68:44733
I20240502 21:25:31.853421 31558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.68:44733 every 8 connection(s)
I20240502 21:25:31.854939 31428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:25:31.856091 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 31428
I20240502 21:25:31.856312 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 21:25:31.860327 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.69:0
--local_ip_for_outbound_sockets=127.29.82.69
--webserver_interface=127.29.82.69
--webserver_port=0
--tserver_master_addrs=127.29.82.126:41665
--builtin_ntp_servers=127.29.82.84:38471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:25:31.871726 31559 heartbeater.cc:347] Connected to a master server at 127.29.82.126:41665
I20240502 21:25:31.872061 31559 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:31.873214 31559 heartbeater.cc:511] Master 127.29.82.126:41665 requested a full tablet report, sending...
I20240502 21:25:31.875069 30957 ts_manager.cc:194] Registered new tserver with Master: 09ad321209604f9eb2c06cbfcbb8bfdf (127.29.82.68:44733)
I20240502 21:25:31.875906 30957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.68:45329
I20240502 21:25:31.951007 31289 heartbeater.cc:503] Master 127.29.82.126:41665 was elected leader, sending a full tablet report...
W20240502 21:25:32.195081 31562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:32.195377 31562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:32.195524 31562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:32.202970 31562 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:25:32.203114 31562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:32.203281 31562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.69
I20240502 21:25:32.212491 31562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:38471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.29.82.69
--webserver_port=0
--tserver_master_addrs=127.29.82.126:41665
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.69
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:32.213891 31562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:32.215683 31562 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:32.228040 31568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:32.228547 31569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:32.229246 31571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:32.229853 31562 server_base.cc:1034] running on GCE node
I20240502 21:25:32.230446 31562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:32.231407 31562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:32.232668 31562 hybrid_clock.cc:648] HybridClock initialized: now 1714685132232605 us; error 47 us; skew 500 ppm
I20240502 21:25:32.233021 31562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:32.235416 31562 webserver.cc:460] Webserver started at http://127.29.82.69:45177/ using document root <none> and password file <none>
I20240502 21:25:32.236222 31562 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:32.236349 31562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:32.236692 31562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:32.239449 31562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "8bf95316ff0443618f356f1414edc237"
format_stamp: "Formatted at 2024-05-02 21:25:32 on dist-test-slave-xzsm"
I20240502 21:25:32.240257 31562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "8bf95316ff0443618f356f1414edc237"
format_stamp: "Formatted at 2024-05-02 21:25:32 on dist-test-slave-xzsm"
I20240502 21:25:32.245260 31562 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 21:25:32.248598 31578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:32.249905 31562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 21:25:32.250057 31562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "8bf95316ff0443618f356f1414edc237"
format_stamp: "Formatted at 2024-05-02 21:25:32 on dist-test-slave-xzsm"
I20240502 21:25:32.250259 31562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:32.266288 31562 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:32.267123 31562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:32.267377 31562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:32.268337 31562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:32.269814 31562 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:32.269915 31562 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:32.270032 31562 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:32.270117 31562 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:32.304683 31562 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.69:46349
I20240502 21:25:32.304728 31693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.69:46349 every 8 connection(s)
I20240502 21:25:32.306046 31562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:25:32.312779 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 31562
I20240502 21:25:32.312929 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 21:25:32.319712 31694 heartbeater.cc:347] Connected to a master server at 127.29.82.126:41665
I20240502 21:25:32.320058 31694 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:32.320818 31694 heartbeater.cc:511] Master 127.29.82.126:41665 requested a full tablet report, sending...
I20240502 21:25:32.322305 30957 ts_manager.cc:194] Registered new tserver with Master: 8bf95316ff0443618f356f1414edc237 (127.29.82.69:46349)
I20240502 21:25:32.323401 30957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.69:37379
I20240502 21:25:32.329025 30025 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 21:25:32.344691 30957 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33960:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 21:25:32.392767 31358 tablet_service.cc:1467] Processing CreateTablet for tablet d1b7b813ba994856b849d287d16443d3 (DEFAULT_TABLE table=TestTable [id=a1d1a4358bf74dea8243198d7f1268d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:32.392767 31493 tablet_service.cc:1467] Processing CreateTablet for tablet d1b7b813ba994856b849d287d16443d3 (DEFAULT_TABLE table=TestTable [id=a1d1a4358bf74dea8243198d7f1268d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:32.394347 31358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1b7b813ba994856b849d287d16443d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:32.394372 31493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1b7b813ba994856b849d287d16443d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:32.394320 31221 tablet_service.cc:1467] Processing CreateTablet for tablet d1b7b813ba994856b849d287d16443d3 (DEFAULT_TABLE table=TestTable [id=a1d1a4358bf74dea8243198d7f1268d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:32.395843 31221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1b7b813ba994856b849d287d16443d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:32.405790 31425 heartbeater.cc:503] Master 127.29.82.126:41665 was elected leader, sending a full tablet report...
I20240502 21:25:32.412616 31714 tablet_bootstrap.cc:492] T d1b7b813ba994856b849d287d16443d3 P 09ad321209604f9eb2c06cbfcbb8bfdf: Bootstrap starting.
I20240502 21:25:32.413564 31715 tablet_bootstrap.cc:492] T d1b7b813ba994856b849d287d16443d3 P 4242a19eef2f48b8a11e0f2e8702a3a9: Bootstrap starting.
I20240502 21:25:32.415751 31716 tablet_bootstrap.cc:492] T d1b7b813ba994856b849d287d16443d3 P 2a3cfd9dd6f844e8a57693f00d34c41a: Bootstrap starting.
I20240502 21:25:32.418020 31715 tablet_bootstrap.cc:654] T d1b7b813ba994856b849d287d16443d3 P 4242a19eef2f48b8a11e0f2e8702a3a9: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:32.418021 31714 tablet_bootstrap.cc:654] T d1b7b813ba994856b849d287d16443d3 P 09ad321209604f9eb2c06cbfcbb8bfdf: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:32.418612 31716 tablet_bootstrap.cc:654] T d1b7b813ba994856b849d287d16443d3 P 2a3cfd9dd6f844e8a57693f00d34c41a: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:32.419378 31715 log.cc:824] T d1b7b813ba994856b849d287d16443d3 P 4242a19eef2f48b8a11e0f2e8702a3a9: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:32.419378 31714 log.cc:824] T d1b7b813ba994856b849d287d16443d3 P 09ad321209604f9eb2c06cbfcbb8bfdf: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:32.419700 31716 log.cc:824] T d1b7b813ba994856b849d287d16443d3 P 2a3cfd9dd6f844e8a57693f00d34c41a: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:32.424551 31714 tablet_bootstrap.cc:492] T d1b7b813ba994856b849d287d16443d3 P 09ad321209604f9eb2c06cbfcbb8bfdf: No bootstrap required, opened a new log
I20240502 21:25:32.424876 31714 ts_tablet_manager.cc:1367] T d1b7b813ba994856b849d287d16443d3 P 09ad321209604f9eb2c06cbfcbb8bfdf: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.008s
I20240502 21:25:32.425787 31715 tablet_bootstrap.cc:492] T d1b7b813ba994856b849d287d16443d3 P 4242a19eef2f48b8a11e0f2e8702a3a9: No bootstrap required, opened a new log
I20240502 21:25:32.425962 31716 tablet_bootstrap.cc:492] T d1b7b813ba994856b849d287d16443d3 P 2a3cfd9dd6f844e8a57693f00d34c41a: No bootstrap required, opened a new log
I20240502 21:25:32.426059 31715 ts_tablet_manager.cc:1367] T d1b7b813ba994856b849d287d16443d3 P 4242a19eef2f48b8a11e0f2e8702a3a9: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.010s
I20240502 21:25:32.426239 31716 ts_tablet_manager.cc:1367] T d1b7b813ba994856b849d287d16443d3 P 2a3cfd9dd6f844e8a57693f00d34c41a: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240502 21:25:32.432623 31715 raft_consensus.cc:348] T d1b7b813ba994856b849d287d16443d3 P 4242a19eef2f48b8a11e0f2e8702a3a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4242a19eef2f48b8a11e0f2e8702a3a9" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39693 } } peers { permanent_uuid: "2a3cfd9dd6f844e8a57693f00d34c41a" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37759 } } peers { permanent_uuid: "09ad321209604f9eb2c06cbfcbb8bfdf" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 44733 } }
I20240502 21:25:32.433030 31715 raft_consensus.cc:374] T d1b7b813ba994856b849d287d16443d3 P 4242a19eef2f48b8a11e0f2e8702a3a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:32.433142 31715 raft_consensus.cc:729] T d1b7b813ba994856b849d287d16443d3 P 4242a19eef2f48b8a11e0f2e8702a3a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4242a19eef2f48b8a11e0f2e8702a3a9, State: Initialized, Role: FOLLOWER
I20240502 21:25:32.433796 31715 consensus_queue.cc:260] T d1b7b813ba994856b849d287d16443d3 P 4242a19eef2f48b8a11e0f2e8702a3a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4242a19eef2f48b8a11e0f2e8702a3a9" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39693 } } peers { permanent_uuid: "2a3cfd9dd6f844e8a57693f00d34c41a" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37759 } } peers { permanent_uuid: "09ad321209604f9eb2c06cbfcbb8bfdf" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 44733 } }
I20240502 21:25:32.434278 31714 raft_consensus.cc:348] T d1b7b813ba994856b849d287d16443d3 P 09ad321209604f9eb2c06cbfcbb8bfdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4242a19eef2f48b8a11e0f2e8702a3a9" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39693 } } peers { permanent_uuid: "2a3cfd9dd6f844e8a57693f00d34c41a" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37759 } } peers { permanent_uuid: "09ad321209604f9eb2c06cbfcbb8bfdf" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 44733 } }
I20240502 21:25:32.434682 31714 raft_consensus.cc:374] T d1b7b813ba994856b849d287d16443d3 P 09ad321209604f9eb2c06cbfcbb8bfdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:32.434800 31714 raft_consensus.cc:729] T d1b7b813ba994856b849d287d16443d3 P 09ad321209604f9eb2c06cbfcbb8bfdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ad321209604f9eb2c06cbfcbb8bfdf, State: Initialized, Role: FOLLOWER
I20240502 21:25:32.435007 31716 raft_consensus.cc:348] T d1b7b813ba994856b849d287d16443d3 P 2a3cfd9dd6f844e8a57693f00d34c41a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4242a19eef2f48b8a11e0f2e8702a3a9" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39693 } } peers { permanent_uuid: "2a3cfd9dd6f844e8a57693f00d34c41a" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37759 } } peers { permanent_uuid: "09ad321209604f9eb2c06cbfcbb8bfdf" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 44733 } }
I20240502 21:25:32.435411 31716 raft_consensus.cc:374] T d1b7b813ba994856b849d287d16443d3 P 2a3cfd9dd6f844e8a57693f00d34c41a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:32.435511 31716 raft_consensus.cc:729] T d1b7b813ba994856b849d287d16443d3 P 2a3cfd9dd6f844e8a57693f00d34c41a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a3cfd9dd6f844e8a57693f00d34c41a, State: Initialized, Role: FOLLOWER
I20240502 21:25:32.435619 31715 ts_tablet_manager.cc:1398] T d1b7b813ba994856b849d287d16443d3 P 4242a19eef2f48b8a11e0f2e8702a3a9: Time spent starting tablet: real 0.009s user 0.008s sys 0.002s
I20240502 21:25:32.435633 31714 consensus_queue.cc:260] T d1b7b813ba994856b849d287d16443d3 P 09ad321209604f9eb2c06cbfcbb8bfdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4242a19eef2f48b8a11e0f2e8702a3a9" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39693 } } peers { permanent_uuid: "2a3cfd9dd6f844e8a57693f00d34c41a" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37759 } } peers { permanent_uuid: "09ad321209604f9eb2c06cbfcbb8bfdf" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 44733 } }
I20240502 21:25:32.436159 31716 consensus_queue.cc:260] T d1b7b813ba994856b849d287d16443d3 P 2a3cfd9dd6f844e8a57693f00d34c41a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4242a19eef2f48b8a11e0f2e8702a3a9" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39693 } } peers { permanent_uuid: "2a3cfd9dd6f844e8a57693f00d34c41a" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37759 } } peers { permanent_uuid: "09ad321209604f9eb2c06cbfcbb8bfdf" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 44733 } }
I20240502 21:25:32.440913 31559 heartbeater.cc:503] Master 127.29.82.126:41665 was elected leader, sending a full tablet report...
I20240502 21:25:32.440999 31716 ts_tablet_manager.cc:1398] T d1b7b813ba994856b849d287d16443d3 P 2a3cfd9dd6f844e8a57693f00d34c41a: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20240502 21:25:32.441337 31714 ts_tablet_manager.cc:1398] T d1b7b813ba994856b849d287d16443d3 P 09ad321209604f9eb2c06cbfcbb8bfdf: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20240502 21:25:32.529960 31721 raft_consensus.cc:482] T d1b7b813ba994856b849d287d16443d3 P 09ad321209604f9eb2c06cbfcbb8bfdf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:32.530269 31721 raft_consensus.cc:504] T d1b7b813ba994856b849d287d16443d3 P 09ad321209604f9eb2c06cbfcbb8bfdf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4242a19eef2f48b8a11e0f2e8702a3a9" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39693 } } peers { permanent_uuid: "2a3cfd9dd6f844e8a57693f00d34c41a" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37759 } } peers { permanent_uuid: "09ad321209604f9eb2c06cbfcbb8bfdf" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 44733 } }
I20240502 21:25:32.532145 31721 leader_election.cc:288] T d1b7b813ba994856b849d287d16443d3 P 09ad321209604f9eb2c06cbfcbb8bfdf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4242a19eef2f48b8a11e0f2e8702a3a9 (127.29.82.66:39693), 2a3cfd9dd6f844e8a57693f00d34c41a (127.29.82.67:37759)
I20240502 21:25:32.540683 31243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1b7b813ba994856b849d287d16443d3" candidate_uuid: "09ad321209604f9eb2c06cbfcbb8bfdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4242a19eef2f48b8a11e0f2e8702a3a9" is_pre_election: true
I20240502 21:25:32.540881 31378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1b7b813ba994856b849d287d16443d3" candidate_uuid: "09ad321209604f9eb2c06cbfcbb8bfdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a3cfd9dd6f844e8a57693f00d34c41a" is_pre_election: true
I20240502 21:25:32.541170 31243 raft_consensus.cc:2454] T d1b7b813ba994856b849d287d16443d3 P 4242a19eef2f48b8a11e0f2e8702a3a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ad321209604f9eb2c06cbfcbb8bfdf in term 0.
I20240502 21:25:32.541347 31378 raft_consensus.cc:2454] T d1b7b813ba994856b849d287d16443d3 P 2a3cfd9dd6f844e8a57693f00d34c41a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ad321209604f9eb2c06cbfcbb8bfdf in term 0.
I20240502 21:25:32.541800 31447 leader_election.cc:302] T d1b7b813ba994856b849d287d16443d3 P 09ad321209604f9eb2c06cbfcbb8bfdf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ad321209604f9eb2c06cbfcbb8bfdf, 4242a19eef2f48b8a11e0f2e8702a3a9; no voters:
I20240502 21:25:32.542160 31721 raft_consensus.cc:2781] T d1b7b813ba994856b849d287d16443d3 P 09ad321209604f9eb2c06cbfcbb8bfdf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:32.542309 31721 raft_consensus.cc:482] T d1b7b813ba994856b849d287d16443d3 P 09ad321209604f9eb2c06cbfcbb8bfdf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:32.542420 31721 raft_consensus.cc:3037] T d1b7b813ba994856b849d287d16443d3 P 09ad321209604f9eb2c06cbfcbb8bfdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:32.544549 31721 raft_consensus.cc:504] T d1b7b813ba994856b849d287d16443d3 P 09ad321209604f9eb2c06cbfcbb8bfdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4242a19eef2f48b8a11e0f2e8702a3a9" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39693 } } peers { permanent_uuid: "2a3cfd9dd6f844e8a57693f00d34c41a" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37759 } } peers { permanent_uuid: "09ad321209604f9eb2c06cbfcbb8bfdf" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 44733 } }
I20240502 21:25:32.545143 31721 leader_election.cc:288] T d1b7b813ba994856b849d287d16443d3 P 09ad321209604f9eb2c06cbfcbb8bfdf [CANDIDATE]: Term 1 election: Requested vote from peers 4242a19eef2f48b8a11e0f2e8702a3a9 (127.29.82.66:39693), 2a3cfd9dd6f844e8a57693f00d34c41a (127.29.82.67:37759)
I20240502 21:25:32.545598 31243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1b7b813ba994856b849d287d16443d3" candidate_uuid: "09ad321209604f9eb2c06cbfcbb8bfdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4242a19eef2f48b8a11e0f2e8702a3a9"
I20240502 21:25:32.545694 31378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1b7b813ba994856b849d287d16443d3" candidate_uuid: "09ad321209604f9eb2c06cbfcbb8bfdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a3cfd9dd6f844e8a57693f00d34c41a"
I20240502 21:25:32.545754 31243 raft_consensus.cc:3037] T d1b7b813ba994856b849d287d16443d3 P 4242a19eef2f48b8a11e0f2e8702a3a9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:32.545881 31378 raft_consensus.cc:3037] T d1b7b813ba994856b849d287d16443d3 P 2a3cfd9dd6f844e8a57693f00d34c41a [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:32.547845 31243 raft_consensus.cc:2454] T d1b7b813ba994856b849d287d16443d3 P 4242a19eef2f48b8a11e0f2e8702a3a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ad321209604f9eb2c06cbfcbb8bfdf in term 1.
I20240502 21:25:32.547953 31378 raft_consensus.cc:2454] T d1b7b813ba994856b849d287d16443d3 P 2a3cfd9dd6f844e8a57693f00d34c41a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ad321209604f9eb2c06cbfcbb8bfdf in term 1.
I20240502 21:25:32.548229 31447 leader_election.cc:302] T d1b7b813ba994856b849d287d16443d3 P 09ad321209604f9eb2c06cbfcbb8bfdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ad321209604f9eb2c06cbfcbb8bfdf, 4242a19eef2f48b8a11e0f2e8702a3a9; no voters:
I20240502 21:25:32.548512 31721 raft_consensus.cc:2781] T d1b7b813ba994856b849d287d16443d3 P 09ad321209604f9eb2c06cbfcbb8bfdf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:32.548920 31721 raft_consensus.cc:686] T d1b7b813ba994856b849d287d16443d3 P 09ad321209604f9eb2c06cbfcbb8bfdf [term 1 LEADER]: Becoming Leader. State: Replica: 09ad321209604f9eb2c06cbfcbb8bfdf, State: Running, Role: LEADER
I20240502 21:25:32.549455 31721 consensus_queue.cc:237] T d1b7b813ba994856b849d287d16443d3 P 09ad321209604f9eb2c06cbfcbb8bfdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4242a19eef2f48b8a11e0f2e8702a3a9" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39693 } } peers { permanent_uuid: "2a3cfd9dd6f844e8a57693f00d34c41a" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37759 } } peers { permanent_uuid: "09ad321209604f9eb2c06cbfcbb8bfdf" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 44733 } }
I20240502 21:25:32.554049 30957 catalog_manager.cc:5484] T d1b7b813ba994856b849d287d16443d3 P 09ad321209604f9eb2c06cbfcbb8bfdf reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ad321209604f9eb2c06cbfcbb8bfdf (127.29.82.68). New cstate: current_term: 1 leader_uuid: "09ad321209604f9eb2c06cbfcbb8bfdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4242a19eef2f48b8a11e0f2e8702a3a9" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39693 } } peers { permanent_uuid: "2a3cfd9dd6f844e8a57693f00d34c41a" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37759 } } peers { permanent_uuid: "09ad321209604f9eb2c06cbfcbb8bfdf" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 44733 } } }
I20240502 21:25:32.595558 30025 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 21:25:32.597775 30025 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 9d3ce5eec771415a89fd7390c6ddaa7a to finish bootstrapping
W20240502 21:25:32.607283 31560 tablet.cc:2369] T d1b7b813ba994856b849d287d16443d3 P 09ad321209604f9eb2c06cbfcbb8bfdf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:32.607481 30025 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4242a19eef2f48b8a11e0f2e8702a3a9 to finish bootstrapping
I20240502 21:25:32.615123 30025 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2a3cfd9dd6f844e8a57693f00d34c41a to finish bootstrapping
I20240502 21:25:32.623143 30025 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 09ad321209604f9eb2c06cbfcbb8bfdf to finish bootstrapping
I20240502 21:25:32.631064 30025 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 8bf95316ff0443618f356f1414edc237 to finish bootstrapping
W20240502 21:25:32.639428 31426 tablet.cc:2369] T d1b7b813ba994856b849d287d16443d3 P 2a3cfd9dd6f844e8a57693f00d34c41a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:32.640218 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 31018
I20240502 21:25:32.641280 31146 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:25:32.682971 31290 tablet.cc:2369] T d1b7b813ba994856b849d287d16443d3 P 4242a19eef2f48b8a11e0f2e8702a3a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:32.722703 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 31018
I20240502 21:25:33.037576 31721 consensus_queue.cc:1035] T d1b7b813ba994856b849d287d16443d3 P 09ad321209604f9eb2c06cbfcbb8bfdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4242a19eef2f48b8a11e0f2e8702a3a9" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:33.046242 31725 consensus_queue.cc:1035] T d1b7b813ba994856b849d287d16443d3 P 09ad321209604f9eb2c06cbfcbb8bfdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a3cfd9dd6f844e8a57693f00d34c41a" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:33.272951 31493 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:33.281749 31626 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:33.282585 31358 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:33.296613 31221 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:33.325892 31694 heartbeater.cc:503] Master 127.29.82.126:41665 was elected leader, sending a full tablet report...
I20240502 21:25:33.430613 31493 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:33.439610 31358 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:33.440933 31626 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:33.451507 31221 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:33.612174 31493 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:33.614327 31221 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:33.619889 31626 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:33.625578 31358 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:34.271349 31358 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:34.273034 31626 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:34.275893 31493 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:34.284205 31221 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:34.421363 31493 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:34.429994 31358 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:34.436116 31221 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:34.438308 31626 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:34.615278 31626 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:34.620105 31493 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:34.620648 31358 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:34.621090 31221 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:34.777154 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 31156
I20240502 21:25:34.778689 31283 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:34.879503 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 31156
I20240502 21:25:34.900913 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 31292
I20240502 21:25:34.901995 31418 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:35.004053 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 31292
I20240502 21:25:35.025236 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 31428
I20240502 21:25:35.026366 31554 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:35.130419 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 31428
I20240502 21:25:35.151710 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 31562
I20240502 21:25:35.152740 31689 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:35.248529 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 31562
I20240502 21:25:35.267613 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 30925
I20240502 21:25:35.268777 30989 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:35.390617 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 30925
2024-05-02T21:25:35Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/0 (5962 ms)
[----------] 1 test from RebalanceStartCriteriaTest (5962 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/0
I20240502 21:25:35.436439 30025 test_util.cc:268] Using random seed: 1046887033
I20240502 21:25:35.438371 30025 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:25:35.438436 30025 ts_itest-base.cc:113] --------------
I20240502 21:25:35.438478 30025 ts_itest-base.cc:114] 7 tablet servers
I20240502 21:25:35.438524 30025 ts_itest-base.cc:115] 1 replicas per TS
I20240502 21:25:35.438567 30025 ts_itest-base.cc:116] --------------
2024-05-02T21:25:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:25:35Z Disabled control of system clock
I20240502 21:25:35.463593 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.82.126:44793
--webserver_interface=127.29.82.126
--webserver_port=0
--builtin_ntp_servers=127.29.82.84:43193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.82.126:44793
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240502 21:25:35.802204 31948 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 21:25:35.802583 31948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:35.802695 31948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:35.802814 31948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:35.810914 31948 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:25:35.811053 31948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 21:25:35.811162 31948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:35.811231 31948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 21:25:35.811321 31948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 21:25:35.821281 31948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:43193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.29.82.126:44793
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.82.126:44793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.29.82.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:35.822876 31948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:35.824779 31948 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:35.838383 31955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:35.839641 31954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:35.839684 31957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:35.840673 31948 server_base.cc:1034] running on GCE node
I20240502 21:25:35.841542 31948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:35.843039 31948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:35.844274 31948 hybrid_clock.cc:648] HybridClock initialized: now 1714685135844212 us; error 41 us; skew 500 ppm
I20240502 21:25:35.844631 31948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:35.852716 31948 webserver.cc:460] Webserver started at http://127.29.82.126:39505/ using document root <none> and password file <none>
I20240502 21:25:35.853518 31948 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:35.853637 31948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:35.853922 31948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:35.856709 31948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3efdd8a7bacc4111a711c88a6c613c1b"
format_stamp: "Formatted at 2024-05-02 21:25:35 on dist-test-slave-xzsm"
I20240502 21:25:35.857436 31948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3efdd8a7bacc4111a711c88a6c613c1b"
format_stamp: "Formatted at 2024-05-02 21:25:35 on dist-test-slave-xzsm"
I20240502 21:25:35.862637 31948 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 21:25:35.866292 31963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:35.867774 31948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240502 21:25:35.867947 31948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3efdd8a7bacc4111a711c88a6c613c1b"
format_stamp: "Formatted at 2024-05-02 21:25:35 on dist-test-slave-xzsm"
I20240502 21:25:35.868131 31948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:35.886248 31948 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:35.887113 31948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:35.887401 31948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:35.906178 31948 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.126:44793
I20240502 21:25:35.906199 32016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.126:44793 every 8 connection(s)
I20240502 21:25:35.907565 31948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 21:25:35.910347 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 31948
I20240502 21:25:35.910574 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 21:25:35.912263 32017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:35.926096 32017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3efdd8a7bacc4111a711c88a6c613c1b: Bootstrap starting.
I20240502 21:25:35.929590 32017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3efdd8a7bacc4111a711c88a6c613c1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:35.930577 32017 log.cc:824] T 00000000000000000000000000000000 P 3efdd8a7bacc4111a711c88a6c613c1b: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:35.933025 32017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3efdd8a7bacc4111a711c88a6c613c1b: No bootstrap required, opened a new log
I20240502 21:25:35.940047 32017 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3efdd8a7bacc4111a711c88a6c613c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efdd8a7bacc4111a711c88a6c613c1b" member_type: VOTER last_known_addr { host: "127.29.82.126" port: 44793 } }
I20240502 21:25:35.940380 32017 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3efdd8a7bacc4111a711c88a6c613c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:35.940460 32017 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3efdd8a7bacc4111a711c88a6c613c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdd8a7bacc4111a711c88a6c613c1b, State: Initialized, Role: FOLLOWER
I20240502 21:25:35.941010 32017 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3efdd8a7bacc4111a711c88a6c613c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efdd8a7bacc4111a711c88a6c613c1b" member_type: VOTER last_known_addr { host: "127.29.82.126" port: 44793 } }
I20240502 21:25:35.941206 32017 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3efdd8a7bacc4111a711c88a6c613c1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:35.941310 32017 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3efdd8a7bacc4111a711c88a6c613c1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:35.941427 32017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3efdd8a7bacc4111a711c88a6c613c1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:36.018368 32017 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3efdd8a7bacc4111a711c88a6c613c1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efdd8a7bacc4111a711c88a6c613c1b" member_type: VOTER last_known_addr { host: "127.29.82.126" port: 44793 } }
I20240502 21:25:36.019040 32017 leader_election.cc:302] T 00000000000000000000000000000000 P 3efdd8a7bacc4111a711c88a6c613c1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3efdd8a7bacc4111a711c88a6c613c1b; no voters:
I20240502 21:25:36.019757 32017 leader_election.cc:288] T 00000000000000000000000000000000 P 3efdd8a7bacc4111a711c88a6c613c1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:36.019954 32022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3efdd8a7bacc4111a711c88a6c613c1b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:36.021237 32022 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3efdd8a7bacc4111a711c88a6c613c1b [term 1 LEADER]: Becoming Leader. State: Replica: 3efdd8a7bacc4111a711c88a6c613c1b, State: Running, Role: LEADER
I20240502 21:25:36.021970 32022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3efdd8a7bacc4111a711c88a6c613c1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efdd8a7bacc4111a711c88a6c613c1b" member_type: VOTER last_known_addr { host: "127.29.82.126" port: 44793 } }
I20240502 21:25:36.022147 32017 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3efdd8a7bacc4111a711c88a6c613c1b [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:25:36.025135 32024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3efdd8a7bacc4111a711c88a6c613c1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3efdd8a7bacc4111a711c88a6c613c1b. Latest consensus state: current_term: 1 leader_uuid: "3efdd8a7bacc4111a711c88a6c613c1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efdd8a7bacc4111a711c88a6c613c1b" member_type: VOTER last_known_addr { host: "127.29.82.126" port: 44793 } } }
I20240502 21:25:36.025315 32023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3efdd8a7bacc4111a711c88a6c613c1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3efdd8a7bacc4111a711c88a6c613c1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efdd8a7bacc4111a711c88a6c613c1b" member_type: VOTER last_known_addr { host: "127.29.82.126" port: 44793 } } }
I20240502 21:25:36.025743 32024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3efdd8a7bacc4111a711c88a6c613c1b [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:36.025779 32023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3efdd8a7bacc4111a711c88a6c613c1b [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:36.028714 32033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:25:36.034317 32033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:25:36.044409 32033 catalog_manager.cc:1347] Generated new cluster ID: 47051f93ace544f9a882a9792bf83ff9
I20240502 21:25:36.044541 32033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:25:36.057487 32033 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:25:36.059054 32033 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:25:36.072286 32033 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3efdd8a7bacc4111a711c88a6c613c1b: Generated new TSK 0
I20240502 21:25:36.073496 32033 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:25:36.080134 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.65:0
--local_ip_for_outbound_sockets=127.29.82.65
--webserver_interface=127.29.82.65
--webserver_port=0
--tserver_master_addrs=127.29.82.126:44793
--builtin_ntp_servers=127.29.82.84:43193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 21:25:36.420323 32043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:36.420634 32043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:36.420784 32043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:36.431649 32043 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:25:36.431828 32043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:36.432003 32043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.65
I20240502 21:25:36.441068 32043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:43193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.29.82.65
--webserver_port=0
--tserver_master_addrs=127.29.82.126:44793
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.65
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:36.442373 32043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:36.444111 32043 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:36.457166 32050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:36.458076 32048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:36.459340 32052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:36.460497 32043 server_base.cc:1034] running on GCE node
I20240502 21:25:36.461153 32043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:36.462168 32043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:36.463349 32043 hybrid_clock.cc:648] HybridClock initialized: now 1714685136463298 us; error 46 us; skew 500 ppm
I20240502 21:25:36.463670 32043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:36.465863 32043 webserver.cc:460] Webserver started at http://127.29.82.65:39531/ using document root <none> and password file <none>
I20240502 21:25:36.466766 32043 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:36.466871 32043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:36.467185 32043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:36.469767 32043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "cb8caceef23e475f8304b1ec1175e4b9"
format_stamp: "Formatted at 2024-05-02 21:25:36 on dist-test-slave-xzsm"
I20240502 21:25:36.470518 32043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "cb8caceef23e475f8304b1ec1175e4b9"
format_stamp: "Formatted at 2024-05-02 21:25:36 on dist-test-slave-xzsm"
I20240502 21:25:36.475648 32043 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20240502 21:25:36.478966 32058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:36.480332 32043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:25:36.480727 32043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "cb8caceef23e475f8304b1ec1175e4b9"
format_stamp: "Formatted at 2024-05-02 21:25:36 on dist-test-slave-xzsm"
I20240502 21:25:36.480928 32043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:36.497829 32043 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:36.498658 32043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:36.498936 32043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:36.499914 32043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:36.501442 32043 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:36.501536 32043 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:36.501657 32043 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:36.501747 32043 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:36.537765 32043 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.65:37947
I20240502 21:25:36.537837 32172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.65:37947 every 8 connection(s)
I20240502 21:25:36.539153 32043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 21:25:36.540557 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 32043
I20240502 21:25:36.540829 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 21:25:36.544862 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.66:0
--local_ip_for_outbound_sockets=127.29.82.66
--webserver_interface=127.29.82.66
--webserver_port=0
--tserver_master_addrs=127.29.82.126:44793
--builtin_ntp_servers=127.29.82.84:43193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:25:36.553879 32173 heartbeater.cc:347] Connected to a master server at 127.29.82.126:44793
I20240502 21:25:36.554176 32173 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:36.554875 32173 heartbeater.cc:511] Master 127.29.82.126:44793 requested a full tablet report, sending...
I20240502 21:25:36.557045 31981 ts_manager.cc:194] Registered new tserver with Master: cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65:37947)
I20240502 21:25:36.559372 31981 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.65:40207
W20240502 21:25:36.891963 32177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:36.892274 32177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:36.892369 32177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:36.900453 32177 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:25:36.900591 32177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:36.900724 32177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.66
I20240502 21:25:36.910172 32177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:43193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.29.82.66
--webserver_port=0
--tserver_master_addrs=127.29.82.126:44793
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.66
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:36.911514 32177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:36.913322 32177 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:36.926792 32183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:36.928150 32184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:36.929075 32186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:36.930109 32177 server_base.cc:1034] running on GCE node
I20240502 21:25:36.930721 32177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:36.931726 32177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:36.932940 32177 hybrid_clock.cc:648] HybridClock initialized: now 1714685136932897 us; error 40 us; skew 500 ppm
I20240502 21:25:36.933262 32177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:36.935549 32177 webserver.cc:460] Webserver started at http://127.29.82.66:45437/ using document root <none> and password file <none>
I20240502 21:25:36.936316 32177 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:36.936434 32177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:36.936812 32177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:36.939409 32177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "f6f8b5e2067c4c23b0c89a8c471626f5"
format_stamp: "Formatted at 2024-05-02 21:25:36 on dist-test-slave-xzsm"
I20240502 21:25:36.940248 32177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "f6f8b5e2067c4c23b0c89a8c471626f5"
format_stamp: "Formatted at 2024-05-02 21:25:36 on dist-test-slave-xzsm"
I20240502 21:25:36.945449 32177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 21:25:36.949051 32193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:36.950340 32177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:25:36.950497 32177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f6f8b5e2067c4c23b0c89a8c471626f5"
format_stamp: "Formatted at 2024-05-02 21:25:36 on dist-test-slave-xzsm"
I20240502 21:25:36.950690 32177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:36.970589 32177 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:36.971450 32177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:36.971773 32177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:36.972783 32177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:36.974311 32177 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:36.974417 32177 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:36.974537 32177 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:36.974611 32177 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:37.008358 32177 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.66:39503
I20240502 21:25:37.008414 32308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.66:39503 every 8 connection(s)
I20240502 21:25:37.009709 32177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 21:25:37.016593 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 32177
I20240502 21:25:37.016791 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 21:25:37.020576 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.67:0
--local_ip_for_outbound_sockets=127.29.82.67
--webserver_interface=127.29.82.67
--webserver_port=0
--tserver_master_addrs=127.29.82.126:44793
--builtin_ntp_servers=127.29.82.84:43193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:25:37.023789 32309 heartbeater.cc:347] Connected to a master server at 127.29.82.126:44793
I20240502 21:25:37.024083 32309 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:37.024838 32309 heartbeater.cc:511] Master 127.29.82.126:44793 requested a full tablet report, sending...
I20240502 21:25:37.026196 31981 ts_manager.cc:194] Registered new tserver with Master: f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66:39503)
I20240502 21:25:37.027384 31981 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.66:46049
W20240502 21:25:37.353701 32312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:37.353982 32312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:37.354096 32312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:37.361881 32312 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:25:37.361994 32312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:37.362107 32312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.67
I20240502 21:25:37.371603 32312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:43193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.29.82.67
--webserver_port=0
--tserver_master_addrs=127.29.82.126:44793
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.67
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:37.372967 32312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:37.374823 32312 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:37.388254 32317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:37.389372 32320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:37.390406 32318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:37.391546 32312 server_base.cc:1034] running on GCE node
I20240502 21:25:37.392172 32312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:37.393126 32312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:37.394348 32312 hybrid_clock.cc:648] HybridClock initialized: now 1714685137394297 us; error 47 us; skew 500 ppm
I20240502 21:25:37.394740 32312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:37.397229 32312 webserver.cc:460] Webserver started at http://127.29.82.67:41011/ using document root <none> and password file <none>
I20240502 21:25:37.398012 32312 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:37.398144 32312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:37.398473 32312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:37.401232 32312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b399bf9ec0c544749c7eead22ddfedf6"
format_stamp: "Formatted at 2024-05-02 21:25:37 on dist-test-slave-xzsm"
I20240502 21:25:37.402331 32312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b399bf9ec0c544749c7eead22ddfedf6"
format_stamp: "Formatted at 2024-05-02 21:25:37 on dist-test-slave-xzsm"
I20240502 21:25:37.408036 32312 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20240502 21:25:37.411504 32327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:37.412959 32312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:25:37.413115 32312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b399bf9ec0c544749c7eead22ddfedf6"
format_stamp: "Formatted at 2024-05-02 21:25:37 on dist-test-slave-xzsm"
I20240502 21:25:37.413333 32312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:37.436128 32312 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:37.437134 32312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:37.437412 32312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:37.438608 32312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:37.440250 32312 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:37.440357 32312 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:37.440481 32312 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:37.440572 32312 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:37.473832 32312 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.67:45329
I20240502 21:25:37.473942 32441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.67:45329 every 8 connection(s)
I20240502 21:25:37.475207 32312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:25:37.480535 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 32312
I20240502 21:25:37.480765 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 21:25:37.484242 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.68:0
--local_ip_for_outbound_sockets=127.29.82.68
--webserver_interface=127.29.82.68
--webserver_port=0
--tserver_master_addrs=127.29.82.126:44793
--builtin_ntp_servers=127.29.82.84:43193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:25:37.489620 32442 heartbeater.cc:347] Connected to a master server at 127.29.82.126:44793
I20240502 21:25:37.489979 32442 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:37.490734 32442 heartbeater.cc:511] Master 127.29.82.126:44793 requested a full tablet report, sending...
I20240502 21:25:37.492374 31981 ts_manager.cc:194] Registered new tserver with Master: b399bf9ec0c544749c7eead22ddfedf6 (127.29.82.67:45329)
I20240502 21:25:37.493547 31981 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.67:51279
I20240502 21:25:37.562403 32173 heartbeater.cc:503] Master 127.29.82.126:44793 was elected leader, sending a full tablet report...
W20240502 21:25:37.824047 32445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:37.824378 32445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:37.824539 32445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:37.832173 32445 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:25:37.832310 32445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:37.832466 32445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.68
I20240502 21:25:37.842401 32445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:43193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.29.82.68
--webserver_port=0
--tserver_master_addrs=127.29.82.126:44793
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.68
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:37.843823 32445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:37.845592 32445 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:37.858312 32450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:37.859172 32451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:37.860486 32453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:37.861577 32445 server_base.cc:1034] running on GCE node
I20240502 21:25:37.862171 32445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:37.863122 32445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:37.864326 32445 hybrid_clock.cc:648] HybridClock initialized: now 1714685137864284 us; error 36 us; skew 500 ppm
I20240502 21:25:37.864652 32445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:37.866768 32445 webserver.cc:460] Webserver started at http://127.29.82.68:45327/ using document root <none> and password file <none>
I20240502 21:25:37.867489 32445 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:37.867647 32445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:37.867964 32445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:37.870536 32445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "f58850eb82214ad39880e45777bf51b4"
format_stamp: "Formatted at 2024-05-02 21:25:37 on dist-test-slave-xzsm"
I20240502 21:25:37.871311 32445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "f58850eb82214ad39880e45777bf51b4"
format_stamp: "Formatted at 2024-05-02 21:25:37 on dist-test-slave-xzsm"
I20240502 21:25:37.876600 32445 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20240502 21:25:37.880010 32459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:37.881368 32445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240502 21:25:37.881551 32445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "f58850eb82214ad39880e45777bf51b4"
format_stamp: "Formatted at 2024-05-02 21:25:37 on dist-test-slave-xzsm"
I20240502 21:25:37.881789 32445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:37.907050 32445 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:37.908178 32445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:37.908507 32445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:37.909768 32445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:37.911636 32445 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:37.911748 32445 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:37.911851 32445 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:37.911921 32445 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:37.944917 32445 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.68:33951
I20240502 21:25:37.944970 32574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.68:33951 every 8 connection(s)
I20240502 21:25:37.946239 32445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:25:37.956490 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 32445
I20240502 21:25:37.956676 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 21:25:37.958563 32575 heartbeater.cc:347] Connected to a master server at 127.29.82.126:44793
I20240502 21:25:37.958853 32575 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:37.959775 32575 heartbeater.cc:511] Master 127.29.82.126:44793 requested a full tablet report, sending...
I20240502 21:25:37.960338 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.69:0
--local_ip_for_outbound_sockets=127.29.82.69
--webserver_interface=127.29.82.69
--webserver_port=0
--tserver_master_addrs=127.29.82.126:44793
--builtin_ntp_servers=127.29.82.84:43193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:25:37.961220 31980 ts_manager.cc:194] Registered new tserver with Master: f58850eb82214ad39880e45777bf51b4 (127.29.82.68:33951)
I20240502 21:25:37.962047 31980 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.68:41515
I20240502 21:25:38.030603 32309 heartbeater.cc:503] Master 127.29.82.126:44793 was elected leader, sending a full tablet report...
W20240502 21:25:38.284811 32578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:38.285115 32578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:38.285274 32578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:38.292804 32578 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:25:38.292937 32578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:38.293100 32578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.69
I20240502 21:25:38.301805 32578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:43193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.29.82.69
--webserver_port=0
--tserver_master_addrs=127.29.82.126:44793
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.69
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:38.303161 32578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:38.304967 32578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:38.317919 32584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:38.319103 32585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:38.320082 32587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:38.321048 32578 server_base.cc:1034] running on GCE node
I20240502 21:25:38.321681 32578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:38.322728 32578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:38.323954 32578 hybrid_clock.cc:648] HybridClock initialized: now 1714685138323889 us; error 56 us; skew 500 ppm
I20240502 21:25:38.324287 32578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:38.326592 32578 webserver.cc:460] Webserver started at http://127.29.82.69:33659/ using document root <none> and password file <none>
I20240502 21:25:38.327364 32578 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:38.327473 32578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:38.327818 32578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:38.330407 32578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "21e73dc1a0eb486db9717fa320347e1c"
format_stamp: "Formatted at 2024-05-02 21:25:38 on dist-test-slave-xzsm"
I20240502 21:25:38.331164 32578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "21e73dc1a0eb486db9717fa320347e1c"
format_stamp: "Formatted at 2024-05-02 21:25:38 on dist-test-slave-xzsm"
I20240502 21:25:38.336051 32578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20240502 21:25:38.339270 32594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:38.340605 32578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:25:38.340757 32578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "21e73dc1a0eb486db9717fa320347e1c"
format_stamp: "Formatted at 2024-05-02 21:25:38 on dist-test-slave-xzsm"
I20240502 21:25:38.340945 32578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:38.363020 32578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:38.363833 32578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:38.364109 32578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:38.365015 32578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:38.366446 32578 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:38.366544 32578 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:38.366662 32578 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:38.366744 32578 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:38.402071 32578 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.69:36679
I20240502 21:25:38.402114 32711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.69:36679 every 8 connection(s)
I20240502 21:25:38.403398 32578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:25:38.408547 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 32578
I20240502 21:25:38.408790 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 21:25:38.412277 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.70:0
--local_ip_for_outbound_sockets=127.29.82.70
--webserver_interface=127.29.82.70
--webserver_port=0
--tserver_master_addrs=127.29.82.126:44793
--builtin_ntp_servers=127.29.82.84:43193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:25:38.417964 32712 heartbeater.cc:347] Connected to a master server at 127.29.82.126:44793
I20240502 21:25:38.418260 32712 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:38.419104 32712 heartbeater.cc:511] Master 127.29.82.126:44793 requested a full tablet report, sending...
I20240502 21:25:38.420639 31980 ts_manager.cc:194] Registered new tserver with Master: 21e73dc1a0eb486db9717fa320347e1c (127.29.82.69:36679)
I20240502 21:25:38.421823 31980 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.69:33589
I20240502 21:25:38.496335 32442 heartbeater.cc:503] Master 127.29.82.126:44793 was elected leader, sending a full tablet report...
W20240502 21:25:38.743278 32715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:38.743564 32715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:38.743702 32715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:38.750988 32715 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:25:38.751093 32715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:38.751217 32715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.70
I20240502 21:25:38.759887 32715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:43193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.29.82.70
--webserver_port=0
--tserver_master_addrs=127.29.82.126:44793
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.70
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:38.761111 32715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:38.762830 32715 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:38.775133 32720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:38.776075 32721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:38.777004 32723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:38.778424 32715 server_base.cc:1034] running on GCE node
I20240502 21:25:38.779008 32715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:38.780012 32715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:38.781224 32715 hybrid_clock.cc:648] HybridClock initialized: now 1714685138781170 us; error 41 us; skew 500 ppm
I20240502 21:25:38.781581 32715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:38.783962 32715 webserver.cc:460] Webserver started at http://127.29.82.70:37091/ using document root <none> and password file <none>
I20240502 21:25:38.784727 32715 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:38.784868 32715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:38.785208 32715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:38.788093 32715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "e705f0dda7904364a7453ee6294b3601"
format_stamp: "Formatted at 2024-05-02 21:25:38 on dist-test-slave-xzsm"
I20240502 21:25:38.788875 32715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "e705f0dda7904364a7453ee6294b3601"
format_stamp: "Formatted at 2024-05-02 21:25:38 on dist-test-slave-xzsm"
I20240502 21:25:38.794108 32715 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 21:25:38.797498 32730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:38.798781 32715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:38.798942 32715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "e705f0dda7904364a7453ee6294b3601"
format_stamp: "Formatted at 2024-05-02 21:25:38 on dist-test-slave-xzsm"
I20240502 21:25:38.799134 32715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:38.816763 32715 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:38.817673 32715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:38.817937 32715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:38.818917 32715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:38.820399 32715 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:38.820499 32715 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:38.820628 32715 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:38.820706 32715 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:38.853238 32715 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.70:45931
I20240502 21:25:38.853296 378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.70:45931 every 8 connection(s)
I20240502 21:25:38.854772 32715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 21:25:38.862565 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 32715
I20240502 21:25:38.862767 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 21:25:38.866571 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.71:0
--local_ip_for_outbound_sockets=127.29.82.71
--webserver_interface=127.29.82.71
--webserver_port=0
--tserver_master_addrs=127.29.82.126:44793
--builtin_ntp_servers=127.29.82.84:43193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:25:38.868436 379 heartbeater.cc:347] Connected to a master server at 127.29.82.126:44793
I20240502 21:25:38.868710 379 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:38.869422 379 heartbeater.cc:511] Master 127.29.82.126:44793 requested a full tablet report, sending...
I20240502 21:25:38.870769 31980 ts_manager.cc:194] Registered new tserver with Master: e705f0dda7904364a7453ee6294b3601 (127.29.82.70:45931)
I20240502 21:25:38.871531 31980 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.70:34131
I20240502 21:25:38.964874 32575 heartbeater.cc:503] Master 127.29.82.126:44793 was elected leader, sending a full tablet report...
W20240502 21:25:39.198397 382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:39.198668 382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:39.198808 382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:39.206291 382 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:25:39.206403 382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:39.206562 382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.71
I20240502 21:25:39.215369 382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:43193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.29.82.71
--webserver_port=0
--tserver_master_addrs=127.29.82.126:44793
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.71
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:39.216677 382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:39.218479 382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:39.231295 390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:39.232173 387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:39.233386 388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:39.234514 382 server_base.cc:1034] running on GCE node
I20240502 21:25:39.235083 382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:39.236006 382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:39.237201 382 hybrid_clock.cc:648] HybridClock initialized: now 1714685139237153 us; error 45 us; skew 500 ppm
I20240502 21:25:39.237493 382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:39.239672 382 webserver.cc:460] Webserver started at http://127.29.82.71:44147/ using document root <none> and password file <none>
I20240502 21:25:39.240339 382 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:39.240433 382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:39.240707 382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:39.243094 382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "98ac0bd5300e42c38bee7aae9634c7c9"
format_stamp: "Formatted at 2024-05-02 21:25:39 on dist-test-slave-xzsm"
I20240502 21:25:39.243882 382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "98ac0bd5300e42c38bee7aae9634c7c9"
format_stamp: "Formatted at 2024-05-02 21:25:39 on dist-test-slave-xzsm"
I20240502 21:25:39.248805 382 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240502 21:25:39.252024 396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:39.253273 382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 21:25:39.253403 382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "98ac0bd5300e42c38bee7aae9634c7c9"
format_stamp: "Formatted at 2024-05-02 21:25:39 on dist-test-slave-xzsm"
I20240502 21:25:39.253566 382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:39.275741 382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:39.276532 382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:39.276789 382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:39.277659 382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:39.279112 382 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:39.279202 382 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:39.279287 382 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:39.279351 382 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:39.312741 382 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.71:39493
I20240502 21:25:39.312803 510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.71:39493 every 8 connection(s)
I20240502 21:25:39.314059 382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 21:25:39.315985 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 382
I20240502 21:25:39.316236 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 21:25:39.326951 511 heartbeater.cc:347] Connected to a master server at 127.29.82.126:44793
I20240502 21:25:39.327204 511 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:39.327921 511 heartbeater.cc:511] Master 127.29.82.126:44793 requested a full tablet report, sending...
I20240502 21:25:39.329128 31980 ts_manager.cc:194] Registered new tserver with Master: 98ac0bd5300e42c38bee7aae9634c7c9 (127.29.82.71:39493)
I20240502 21:25:39.329931 31980 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.71:38095
I20240502 21:25:39.332417 30025 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 21:25:39.347872 31980 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51326:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 21:25:39.374560 32107 tablet_service.cc:1467] Processing CreateTablet for tablet 9812cd5d31154807a4f605eb615553f1 (DEFAULT_TABLE table=TestTable [id=ebf8d845a721492e8b62cd5dc012abae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:39.375922 32107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9812cd5d31154807a4f605eb615553f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:39.386039 525 tablet_bootstrap.cc:492] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9: Bootstrap starting.
I20240502 21:25:39.389578 525 tablet_bootstrap.cc:654] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:39.390643 525 log.cc:824] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:39.393069 525 tablet_bootstrap.cc:492] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9: No bootstrap required, opened a new log
I20240502 21:25:39.393316 525 ts_tablet_manager.cc:1367] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 21:25:39.399148 525 raft_consensus.cc:348] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:39.399448 525 raft_consensus.cc:374] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:39.399547 525 raft_consensus.cc:729] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb8caceef23e475f8304b1ec1175e4b9, State: Initialized, Role: FOLLOWER
I20240502 21:25:39.400195 525 consensus_queue.cc:260] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:39.400394 525 raft_consensus.cc:388] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:39.400523 525 raft_consensus.cc:482] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:39.400676 525 raft_consensus.cc:3037] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:39.402967 525 raft_consensus.cc:504] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:39.403476 525 leader_election.cc:302] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb8caceef23e475f8304b1ec1175e4b9; no voters:
I20240502 21:25:39.404143 525 leader_election.cc:288] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:39.404326 527 raft_consensus.cc:2781] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:39.405735 525 ts_tablet_manager.cc:1398] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20240502 21:25:39.406167 527 raft_consensus.cc:686] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Becoming Leader. State: Replica: cb8caceef23e475f8304b1ec1175e4b9, State: Running, Role: LEADER
I20240502 21:25:39.406687 527 consensus_queue.cc:237] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:39.411938 31980 catalog_manager.cc:5484] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "cb8caceef23e475f8304b1ec1175e4b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } }
I20240502 21:25:39.424480 32712 heartbeater.cc:503] Master 127.29.82.126:44793 was elected leader, sending a full tablet report...
I20240502 21:25:39.431613 30025 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 21:25:39.433563 30025 ts_itest-base.cc:233] Waiting for 1 tablets on tserver cb8caceef23e475f8304b1ec1175e4b9 to finish bootstrapping
I20240502 21:25:39.442039 30025 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f6f8b5e2067c4c23b0c89a8c471626f5 to finish bootstrapping
I20240502 21:25:39.451352 30025 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b399bf9ec0c544749c7eead22ddfedf6 to finish bootstrapping
I20240502 21:25:39.459832 30025 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f58850eb82214ad39880e45777bf51b4 to finish bootstrapping
I20240502 21:25:39.468114 30025 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 21e73dc1a0eb486db9717fa320347e1c to finish bootstrapping
I20240502 21:25:39.476490 30025 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e705f0dda7904364a7453ee6294b3601 to finish bootstrapping
I20240502 21:25:39.484515 30025 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 98ac0bd5300e42c38bee7aae9634c7c9 to finish bootstrapping
I20240502 21:25:39.492975 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 32312
I20240502 21:25:39.493992 32437 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:39.583968 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 32312
W20240502 21:25:39.601552 31967 connection.cc:535] server connection from 127.29.82.67:51279 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:25:39.601876 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 32445
I20240502 21:25:39.603040 32570 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:39.687762 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 32445
I20240502 21:25:39.705914 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 32578
I20240502 21:25:39.707088 32707 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:39.788599 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 32578
I20240502 21:25:39.805995 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 32715
I20240502 21:25:39.807107 373 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:39.886950 379 heartbeater.cc:503] Master 127.29.82.126:44793 was elected leader, sending a full tablet report...
I20240502 21:25:39.887720 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 32715
W20240502 21:25:39.905108 31965 connection.cc:535] server connection from 127.29.82.70:34131 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:25:39.905440 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 382
I20240502 21:25:39.906713 506 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:39.987533 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 382
I20240502 21:25:43.757242 31980 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51326:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:25:43.767812 32107 tablet_service.cc:1467] Processing CreateTablet for tablet bc9b263fc4224e8eb8ca3c82a808bce8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7a5ba90921d043e6a70466c331f7ef93]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:43.768321 32107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc9b263fc4224e8eb8ca3c82a808bce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:43.773872 551 tablet_bootstrap.cc:492] T bc9b263fc4224e8eb8ca3c82a808bce8 P cb8caceef23e475f8304b1ec1175e4b9: Bootstrap starting.
I20240502 21:25:43.775669 551 tablet_bootstrap.cc:654] T bc9b263fc4224e8eb8ca3c82a808bce8 P cb8caceef23e475f8304b1ec1175e4b9: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:43.777356 551 tablet_bootstrap.cc:492] T bc9b263fc4224e8eb8ca3c82a808bce8 P cb8caceef23e475f8304b1ec1175e4b9: No bootstrap required, opened a new log
I20240502 21:25:43.776777 32242 tablet_service.cc:1467] Processing CreateTablet for tablet 3843ac68594f43079a1253a877c48065 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7a5ba90921d043e6a70466c331f7ef93]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:43.777578 551 ts_tablet_manager.cc:1367] T bc9b263fc4224e8eb8ca3c82a808bce8 P cb8caceef23e475f8304b1ec1175e4b9: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240502 21:25:43.776820 32241 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1e70e3aa9940f3a6515bc61569565c (DEFAULT_TABLE table=rebalance_test_table_0 [id=7a5ba90921d043e6a70466c331f7ef93]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:43.778529 32242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3843ac68594f43079a1253a877c48065. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:43.778520 551 raft_consensus.cc:348] T bc9b263fc4224e8eb8ca3c82a808bce8 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:43.778697 551 raft_consensus.cc:374] T bc9b263fc4224e8eb8ca3c82a808bce8 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:43.778795 551 raft_consensus.cc:729] T bc9b263fc4224e8eb8ca3c82a808bce8 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb8caceef23e475f8304b1ec1175e4b9, State: Initialized, Role: FOLLOWER
I20240502 21:25:43.779006 32241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f1e70e3aa9940f3a6515bc61569565c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:43.779052 551 consensus_queue.cc:260] T bc9b263fc4224e8eb8ca3c82a808bce8 P cb8caceef23e475f8304b1ec1175e4b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:43.779261 551 raft_consensus.cc:388] T bc9b263fc4224e8eb8ca3c82a808bce8 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:43.779345 551 raft_consensus.cc:482] T bc9b263fc4224e8eb8ca3c82a808bce8 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:43.779451 551 raft_consensus.cc:3037] T bc9b263fc4224e8eb8ca3c82a808bce8 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:43.781217 551 raft_consensus.cc:504] T bc9b263fc4224e8eb8ca3c82a808bce8 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:43.781443 551 leader_election.cc:302] T bc9b263fc4224e8eb8ca3c82a808bce8 P cb8caceef23e475f8304b1ec1175e4b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb8caceef23e475f8304b1ec1175e4b9; no voters:
I20240502 21:25:43.781893 551 leader_election.cc:288] T bc9b263fc4224e8eb8ca3c82a808bce8 P cb8caceef23e475f8304b1ec1175e4b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:43.782056 553 raft_consensus.cc:2781] T bc9b263fc4224e8eb8ca3c82a808bce8 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:43.783128 551 ts_tablet_manager.cc:1398] T bc9b263fc4224e8eb8ca3c82a808bce8 P cb8caceef23e475f8304b1ec1175e4b9: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20240502 21:25:43.783936 553 raft_consensus.cc:686] T bc9b263fc4224e8eb8ca3c82a808bce8 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Becoming Leader. State: Replica: cb8caceef23e475f8304b1ec1175e4b9, State: Running, Role: LEADER
I20240502 21:25:43.784229 553 consensus_queue.cc:237] T bc9b263fc4224e8eb8ca3c82a808bce8 P cb8caceef23e475f8304b1ec1175e4b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:43.786430 31980 catalog_manager.cc:5484] T bc9b263fc4224e8eb8ca3c82a808bce8 P cb8caceef23e475f8304b1ec1175e4b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "cb8caceef23e475f8304b1ec1175e4b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } }
I20240502 21:25:43.794346 558 tablet_bootstrap.cc:492] T 4f1e70e3aa9940f3a6515bc61569565c P f6f8b5e2067c4c23b0c89a8c471626f5: Bootstrap starting.
I20240502 21:25:43.797091 558 tablet_bootstrap.cc:654] T 4f1e70e3aa9940f3a6515bc61569565c P f6f8b5e2067c4c23b0c89a8c471626f5: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:43.798027 558 log.cc:824] T 4f1e70e3aa9940f3a6515bc61569565c P f6f8b5e2067c4c23b0c89a8c471626f5: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:43.800293 558 tablet_bootstrap.cc:492] T 4f1e70e3aa9940f3a6515bc61569565c P f6f8b5e2067c4c23b0c89a8c471626f5: No bootstrap required, opened a new log
I20240502 21:25:43.800561 558 ts_tablet_manager.cc:1367] T 4f1e70e3aa9940f3a6515bc61569565c P f6f8b5e2067c4c23b0c89a8c471626f5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 21:25:43.806099 558 raft_consensus.cc:348] T 4f1e70e3aa9940f3a6515bc61569565c P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:43.806380 558 raft_consensus.cc:374] T 4f1e70e3aa9940f3a6515bc61569565c P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:43.806453 558 raft_consensus.cc:729] T 4f1e70e3aa9940f3a6515bc61569565c P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f8b5e2067c4c23b0c89a8c471626f5, State: Initialized, Role: FOLLOWER
I20240502 21:25:43.807016 558 consensus_queue.cc:260] T 4f1e70e3aa9940f3a6515bc61569565c P f6f8b5e2067c4c23b0c89a8c471626f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:43.807207 558 raft_consensus.cc:388] T 4f1e70e3aa9940f3a6515bc61569565c P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:43.807305 558 raft_consensus.cc:482] T 4f1e70e3aa9940f3a6515bc61569565c P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:43.807425 558 raft_consensus.cc:3037] T 4f1e70e3aa9940f3a6515bc61569565c P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:43.809564 558 raft_consensus.cc:504] T 4f1e70e3aa9940f3a6515bc61569565c P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:43.809979 558 leader_election.cc:302] T 4f1e70e3aa9940f3a6515bc61569565c P f6f8b5e2067c4c23b0c89a8c471626f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6f8b5e2067c4c23b0c89a8c471626f5; no voters:
I20240502 21:25:43.810488 558 leader_election.cc:288] T 4f1e70e3aa9940f3a6515bc61569565c P f6f8b5e2067c4c23b0c89a8c471626f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:43.810631 560 raft_consensus.cc:2781] T 4f1e70e3aa9940f3a6515bc61569565c P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:43.811277 560 raft_consensus.cc:686] T 4f1e70e3aa9940f3a6515bc61569565c P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Becoming Leader. State: Replica: f6f8b5e2067c4c23b0c89a8c471626f5, State: Running, Role: LEADER
I20240502 21:25:43.811918 560 consensus_queue.cc:237] T 4f1e70e3aa9940f3a6515bc61569565c P f6f8b5e2067c4c23b0c89a8c471626f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:43.812162 558 ts_tablet_manager.cc:1398] T 4f1e70e3aa9940f3a6515bc61569565c P f6f8b5e2067c4c23b0c89a8c471626f5: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20240502 21:25:43.812505 558 tablet_bootstrap.cc:492] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5: Bootstrap starting.
I20240502 21:25:43.813836 558 tablet_bootstrap.cc:654] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:43.816084 558 tablet_bootstrap.cc:492] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5: No bootstrap required, opened a new log
I20240502 21:25:43.816222 558 ts_tablet_manager.cc:1367] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240502 21:25:43.816610 31980 catalog_manager.cc:5484] T 4f1e70e3aa9940f3a6515bc61569565c P f6f8b5e2067c4c23b0c89a8c471626f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } }
I20240502 21:25:43.816954 558 raft_consensus.cc:348] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:43.817075 558 raft_consensus.cc:374] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:43.817163 558 raft_consensus.cc:729] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f8b5e2067c4c23b0c89a8c471626f5, State: Initialized, Role: FOLLOWER
I20240502 21:25:43.817395 558 consensus_queue.cc:260] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:43.817574 558 raft_consensus.cc:388] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:43.817662 558 raft_consensus.cc:482] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:43.817768 558 raft_consensus.cc:3037] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:43.819964 558 raft_consensus.cc:504] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:43.820210 558 leader_election.cc:302] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6f8b5e2067c4c23b0c89a8c471626f5; no voters:
I20240502 21:25:43.820410 558 leader_election.cc:288] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:43.820468 560 raft_consensus.cc:2781] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:43.820683 560 raft_consensus.cc:686] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Becoming Leader. State: Replica: f6f8b5e2067c4c23b0c89a8c471626f5, State: Running, Role: LEADER
I20240502 21:25:43.820847 558 ts_tablet_manager.cc:1398] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20240502 21:25:43.820930 560 consensus_queue.cc:237] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:43.823427 31981 catalog_manager.cc:5484] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } }
W20240502 21:25:44.156498 567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:44.156812 567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:44.164269 567 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 113.639 ms
time per row: 0.0568196 ms
I20240502 21:25:44.420464 31977 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51326:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:25:44.430029 32107 tablet_service.cc:1467] Processing CreateTablet for tablet 699c3a3f730248bdacf122126883f2f4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4a48efbb86944c3897b93b4662c377ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:44.430182 32241 tablet_service.cc:1467] Processing CreateTablet for tablet cc8ba66c2b954876891bdded155d2b91 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4a48efbb86944c3897b93b4662c377ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:44.430361 32106 tablet_service.cc:1467] Processing CreateTablet for tablet d484c2aeb0904564997462a634b541d3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4a48efbb86944c3897b93b4662c377ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:44.430622 32241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc8ba66c2b954876891bdded155d2b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:44.430637 32107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 699c3a3f730248bdacf122126883f2f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:44.430972 32106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d484c2aeb0904564997462a634b541d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:44.437769 610 tablet_bootstrap.cc:492] T cc8ba66c2b954876891bdded155d2b91 P f6f8b5e2067c4c23b0c89a8c471626f5: Bootstrap starting.
I20240502 21:25:44.438771 609 tablet_bootstrap.cc:492] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9: Bootstrap starting.
I20240502 21:25:44.439416 610 tablet_bootstrap.cc:654] T cc8ba66c2b954876891bdded155d2b91 P f6f8b5e2067c4c23b0c89a8c471626f5: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:44.440225 609 tablet_bootstrap.cc:654] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:44.441470 610 tablet_bootstrap.cc:492] T cc8ba66c2b954876891bdded155d2b91 P f6f8b5e2067c4c23b0c89a8c471626f5: No bootstrap required, opened a new log
I20240502 21:25:44.441608 610 ts_tablet_manager.cc:1367] T cc8ba66c2b954876891bdded155d2b91 P f6f8b5e2067c4c23b0c89a8c471626f5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:25:44.442239 609 tablet_bootstrap.cc:492] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9: No bootstrap required, opened a new log
I20240502 21:25:44.442382 609 ts_tablet_manager.cc:1367] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:25:44.442440 610 raft_consensus.cc:348] T cc8ba66c2b954876891bdded155d2b91 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:44.442646 610 raft_consensus.cc:374] T cc8ba66c2b954876891bdded155d2b91 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:44.442741 610 raft_consensus.cc:729] T cc8ba66c2b954876891bdded155d2b91 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f8b5e2067c4c23b0c89a8c471626f5, State: Initialized, Role: FOLLOWER
I20240502 21:25:44.442971 610 consensus_queue.cc:260] T cc8ba66c2b954876891bdded155d2b91 P f6f8b5e2067c4c23b0c89a8c471626f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:44.443172 610 raft_consensus.cc:388] T cc8ba66c2b954876891bdded155d2b91 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:44.443171 609 raft_consensus.cc:348] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:44.443272 610 raft_consensus.cc:482] T cc8ba66c2b954876891bdded155d2b91 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:44.443343 609 raft_consensus.cc:374] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:44.443389 610 raft_consensus.cc:3037] T cc8ba66c2b954876891bdded155d2b91 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:44.443447 609 raft_consensus.cc:729] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb8caceef23e475f8304b1ec1175e4b9, State: Initialized, Role: FOLLOWER
I20240502 21:25:44.443719 609 consensus_queue.cc:260] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:44.443926 609 raft_consensus.cc:388] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:44.444018 609 raft_consensus.cc:482] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:44.444125 609 raft_consensus.cc:3037] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:44.446235 610 raft_consensus.cc:504] T cc8ba66c2b954876891bdded155d2b91 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:44.446496 610 leader_election.cc:302] T cc8ba66c2b954876891bdded155d2b91 P f6f8b5e2067c4c23b0c89a8c471626f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6f8b5e2067c4c23b0c89a8c471626f5; no voters:
I20240502 21:25:44.446717 610 leader_election.cc:288] T cc8ba66c2b954876891bdded155d2b91 P f6f8b5e2067c4c23b0c89a8c471626f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:44.446828 561 raft_consensus.cc:2781] T cc8ba66c2b954876891bdded155d2b91 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:44.446802 609 raft_consensus.cc:504] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:44.447203 610 ts_tablet_manager.cc:1398] T cc8ba66c2b954876891bdded155d2b91 P f6f8b5e2067c4c23b0c89a8c471626f5: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:25:44.447340 609 leader_election.cc:302] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb8caceef23e475f8304b1ec1175e4b9; no voters:
I20240502 21:25:44.447350 561 raft_consensus.cc:686] T cc8ba66c2b954876891bdded155d2b91 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Becoming Leader. State: Replica: f6f8b5e2067c4c23b0c89a8c471626f5, State: Running, Role: LEADER
I20240502 21:25:44.447603 609 leader_election.cc:288] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:44.447829 601 raft_consensus.cc:2781] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:44.447736 561 consensus_queue.cc:237] T cc8ba66c2b954876891bdded155d2b91 P f6f8b5e2067c4c23b0c89a8c471626f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:44.449312 609 ts_tablet_manager.cc:1398] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20240502 21:25:44.449631 609 tablet_bootstrap.cc:492] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9: Bootstrap starting.
I20240502 21:25:44.450520 31977 catalog_manager.cc:5484] T cc8ba66c2b954876891bdded155d2b91 P f6f8b5e2067c4c23b0c89a8c471626f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } }
I20240502 21:25:44.451098 609 tablet_bootstrap.cc:654] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:44.453692 609 tablet_bootstrap.cc:492] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9: No bootstrap required, opened a new log
I20240502 21:25:44.454018 609 ts_tablet_manager.cc:1367] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:25:44.454735 609 raft_consensus.cc:348] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:44.454974 609 raft_consensus.cc:374] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:44.455058 609 raft_consensus.cc:729] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb8caceef23e475f8304b1ec1175e4b9, State: Initialized, Role: FOLLOWER
I20240502 21:25:44.455346 609 consensus_queue.cc:260] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:44.455529 609 raft_consensus.cc:388] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:44.455741 609 raft_consensus.cc:482] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:44.455858 609 raft_consensus.cc:3037] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:44.458325 609 raft_consensus.cc:504] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:44.458549 609 leader_election.cc:302] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb8caceef23e475f8304b1ec1175e4b9; no voters:
I20240502 21:25:44.458761 609 leader_election.cc:288] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:44.458828 614 raft_consensus.cc:2781] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:44.459276 609 ts_tablet_manager.cc:1398] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:25:44.459471 614 raft_consensus.cc:686] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Becoming Leader. State: Replica: cb8caceef23e475f8304b1ec1175e4b9, State: Running, Role: LEADER
I20240502 21:25:44.459787 614 consensus_queue.cc:237] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:44.462062 601 raft_consensus.cc:686] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Becoming Leader. State: Replica: cb8caceef23e475f8304b1ec1175e4b9, State: Running, Role: LEADER
I20240502 21:25:44.462324 601 consensus_queue.cc:237] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:44.464329 31977 catalog_manager.cc:5484] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "cb8caceef23e475f8304b1ec1175e4b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } }
I20240502 21:25:44.468310 31977 catalog_manager.cc:5484] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "cb8caceef23e475f8304b1ec1175e4b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } }
W20240502 21:25:44.807747 621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:44.808049 621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:44.815430 621 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 124.748 ms
time per row: 0.0623742 ms
I20240502 21:25:45.082180 31976 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51326:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:25:45.091780 32242 tablet_service.cc:1467] Processing CreateTablet for tablet a80b4f90d4b04e1886e494cf23a6284b (DEFAULT_TABLE table=rebalance_test_table_2 [id=bff79bbad6b54d32b0ee9781a39e2579]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.091781 32241 tablet_service.cc:1467] Processing CreateTablet for tablet 43d4ba4bfe70416bb9c06a3bfe299835 (DEFAULT_TABLE table=rebalance_test_table_2 [id=bff79bbad6b54d32b0ee9781a39e2579]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.092181 32107 tablet_service.cc:1467] Processing CreateTablet for tablet 7ee3c779759a49fd9db84c4f6b73e670 (DEFAULT_TABLE table=rebalance_test_table_2 [id=bff79bbad6b54d32b0ee9781a39e2579]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.092355 32242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a80b4f90d4b04e1886e494cf23a6284b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.092613 32107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ee3c779759a49fd9db84c4f6b73e670. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.092658 32241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43d4ba4bfe70416bb9c06a3bfe299835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.098942 663 tablet_bootstrap.cc:492] T 7ee3c779759a49fd9db84c4f6b73e670 P cb8caceef23e475f8304b1ec1175e4b9: Bootstrap starting.
I20240502 21:25:45.100507 663 tablet_bootstrap.cc:654] T 7ee3c779759a49fd9db84c4f6b73e670 P cb8caceef23e475f8304b1ec1175e4b9: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:45.102329 663 tablet_bootstrap.cc:492] T 7ee3c779759a49fd9db84c4f6b73e670 P cb8caceef23e475f8304b1ec1175e4b9: No bootstrap required, opened a new log
I20240502 21:25:45.102463 663 ts_tablet_manager.cc:1367] T 7ee3c779759a49fd9db84c4f6b73e670 P cb8caceef23e475f8304b1ec1175e4b9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:25:45.103050 663 raft_consensus.cc:348] T 7ee3c779759a49fd9db84c4f6b73e670 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:45.103201 663 raft_consensus.cc:374] T 7ee3c779759a49fd9db84c4f6b73e670 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:45.103299 663 raft_consensus.cc:729] T 7ee3c779759a49fd9db84c4f6b73e670 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb8caceef23e475f8304b1ec1175e4b9, State: Initialized, Role: FOLLOWER
I20240502 21:25:45.103510 663 consensus_queue.cc:260] T 7ee3c779759a49fd9db84c4f6b73e670 P cb8caceef23e475f8304b1ec1175e4b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:45.103710 663 raft_consensus.cc:388] T 7ee3c779759a49fd9db84c4f6b73e670 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:45.103794 663 raft_consensus.cc:482] T 7ee3c779759a49fd9db84c4f6b73e670 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:45.103914 663 raft_consensus.cc:3037] T 7ee3c779759a49fd9db84c4f6b73e670 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:45.105784 663 raft_consensus.cc:504] T 7ee3c779759a49fd9db84c4f6b73e670 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:45.105849 664 tablet_bootstrap.cc:492] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5: Bootstrap starting.
I20240502 21:25:45.106006 663 leader_election.cc:302] T 7ee3c779759a49fd9db84c4f6b73e670 P cb8caceef23e475f8304b1ec1175e4b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb8caceef23e475f8304b1ec1175e4b9; no voters:
I20240502 21:25:45.106194 663 leader_election.cc:288] T 7ee3c779759a49fd9db84c4f6b73e670 P cb8caceef23e475f8304b1ec1175e4b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:45.106246 601 raft_consensus.cc:2781] T 7ee3c779759a49fd9db84c4f6b73e670 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:45.106446 601 raft_consensus.cc:686] T 7ee3c779759a49fd9db84c4f6b73e670 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Becoming Leader. State: Replica: cb8caceef23e475f8304b1ec1175e4b9, State: Running, Role: LEADER
I20240502 21:25:45.106729 601 consensus_queue.cc:237] T 7ee3c779759a49fd9db84c4f6b73e670 P cb8caceef23e475f8304b1ec1175e4b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:45.107360 664 tablet_bootstrap.cc:654] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:45.109779 31976 catalog_manager.cc:5484] T 7ee3c779759a49fd9db84c4f6b73e670 P cb8caceef23e475f8304b1ec1175e4b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "cb8caceef23e475f8304b1ec1175e4b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } }
I20240502 21:25:45.109916 663 ts_tablet_manager.cc:1398] T 7ee3c779759a49fd9db84c4f6b73e670 P cb8caceef23e475f8304b1ec1175e4b9: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20240502 21:25:45.110173 664 tablet_bootstrap.cc:492] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5: No bootstrap required, opened a new log
I20240502 21:25:45.110311 664 ts_tablet_manager.cc:1367] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 21:25:45.111052 664 raft_consensus.cc:348] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:45.111256 664 raft_consensus.cc:374] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:45.111361 664 raft_consensus.cc:729] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f8b5e2067c4c23b0c89a8c471626f5, State: Initialized, Role: FOLLOWER
I20240502 21:25:45.111634 664 consensus_queue.cc:260] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:45.111873 664 raft_consensus.cc:388] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:45.111980 664 raft_consensus.cc:482] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:45.112110 664 raft_consensus.cc:3037] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:45.114207 664 raft_consensus.cc:504] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:45.114432 664 leader_election.cc:302] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6f8b5e2067c4c23b0c89a8c471626f5; no voters:
I20240502 21:25:45.114604 664 leader_election.cc:288] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:45.114699 561 raft_consensus.cc:2781] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:45.115886 561 raft_consensus.cc:686] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Becoming Leader. State: Replica: f6f8b5e2067c4c23b0c89a8c471626f5, State: Running, Role: LEADER
I20240502 21:25:45.116217 561 consensus_queue.cc:237] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:45.116586 664 ts_tablet_manager.cc:1398] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20240502 21:25:45.116833 664 tablet_bootstrap.cc:492] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5: Bootstrap starting.
I20240502 21:25:45.118710 664 tablet_bootstrap.cc:654] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:45.119560 31976 catalog_manager.cc:5484] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } }
I20240502 21:25:45.120481 664 tablet_bootstrap.cc:492] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5: No bootstrap required, opened a new log
I20240502 21:25:45.120640 664 ts_tablet_manager.cc:1367] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240502 21:25:45.121315 664 raft_consensus.cc:348] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:45.121500 664 raft_consensus.cc:374] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:45.121594 664 raft_consensus.cc:729] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f8b5e2067c4c23b0c89a8c471626f5, State: Initialized, Role: FOLLOWER
I20240502 21:25:45.121837 664 consensus_queue.cc:260] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:45.122031 664 raft_consensus.cc:388] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:45.122135 664 raft_consensus.cc:482] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:45.122231 664 raft_consensus.cc:3037] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:45.124379 664 raft_consensus.cc:504] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:45.124631 664 leader_election.cc:302] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6f8b5e2067c4c23b0c89a8c471626f5; no voters:
I20240502 21:25:45.124843 664 leader_election.cc:288] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:45.124922 669 raft_consensus.cc:2781] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:45.125186 669 raft_consensus.cc:686] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Becoming Leader. State: Replica: f6f8b5e2067c4c23b0c89a8c471626f5, State: Running, Role: LEADER
I20240502 21:25:45.125483 664 ts_tablet_manager.cc:1398] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20240502 21:25:45.125533 669 consensus_queue.cc:237] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:45.128085 31976 catalog_manager.cc:5484] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } }
W20240502 21:25:45.458312 674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:45.458647 674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:45.465755 674 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 135.488 ms
time per row: 0.0677439 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:25:46.721092 31978 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51326:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:25:46.730434 32107 tablet_service.cc:1467] Processing CreateTablet for tablet 9873952de1224793b8b7f17ef92295b7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=74a97a8ddca948248cc5d22c9b96616c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:46.730595 32241 tablet_service.cc:1467] Processing CreateTablet for tablet e60fc6b2f53444869f20be78f7c7cd6e (DEFAULT_TABLE table=rebalance_test_table_3 [id=74a97a8ddca948248cc5d22c9b96616c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:46.730928 32107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9873952de1224793b8b7f17ef92295b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:46.730859 32106 tablet_service.cc:1467] Processing CreateTablet for tablet 4919235c52834df5b3924d93391f171a (DEFAULT_TABLE table=rebalance_test_table_3 [id=74a97a8ddca948248cc5d22c9b96616c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:46.731057 32241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e60fc6b2f53444869f20be78f7c7cd6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:46.731216 32106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4919235c52834df5b3924d93391f171a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:46.737382 720 tablet_bootstrap.cc:492] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5: Bootstrap starting.
I20240502 21:25:46.738004 721 tablet_bootstrap.cc:492] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9: Bootstrap starting.
I20240502 21:25:46.739369 720 tablet_bootstrap.cc:654] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:46.739625 721 tablet_bootstrap.cc:654] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:46.741420 720 tablet_bootstrap.cc:492] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5: No bootstrap required, opened a new log
I20240502 21:25:46.741447 721 tablet_bootstrap.cc:492] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9: No bootstrap required, opened a new log
I20240502 21:25:46.741569 720 ts_tablet_manager.cc:1367] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:25:46.741600 721 ts_tablet_manager.cc:1367] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:25:46.742308 721 raft_consensus.cc:348] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:46.742308 720 raft_consensus.cc:348] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:46.742502 721 raft_consensus.cc:374] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:46.742502 720 raft_consensus.cc:374] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:46.742599 720 raft_consensus.cc:729] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f8b5e2067c4c23b0c89a8c471626f5, State: Initialized, Role: FOLLOWER
I20240502 21:25:46.742609 721 raft_consensus.cc:729] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb8caceef23e475f8304b1ec1175e4b9, State: Initialized, Role: FOLLOWER
I20240502 21:25:46.742830 721 consensus_queue.cc:260] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:46.742830 720 consensus_queue.cc:260] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:46.743029 721 raft_consensus.cc:388] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:46.743036 720 raft_consensus.cc:388] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:46.743119 721 raft_consensus.cc:482] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:46.743124 720 raft_consensus.cc:482] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:46.743235 720 raft_consensus.cc:3037] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.743235 721 raft_consensus.cc:3037] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.745879 721 raft_consensus.cc:504] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:46.745867 720 raft_consensus.cc:504] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:46.746129 720 leader_election.cc:302] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6f8b5e2067c4c23b0c89a8c471626f5; no voters:
I20240502 21:25:46.746125 721 leader_election.cc:302] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb8caceef23e475f8304b1ec1175e4b9; no voters:
I20240502 21:25:46.746564 721 leader_election.cc:288] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:46.746678 725 raft_consensus.cc:2781] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:46.747108 720 leader_election.cc:288] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:46.747221 724 raft_consensus.cc:2781] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:46.747607 721 ts_tablet_manager.cc:1398] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 21:25:46.748015 721 tablet_bootstrap.cc:492] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9: Bootstrap starting.
I20240502 21:25:46.750223 721 tablet_bootstrap.cc:654] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:46.750320 725 raft_consensus.cc:686] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Becoming Leader. State: Replica: cb8caceef23e475f8304b1ec1175e4b9, State: Running, Role: LEADER
I20240502 21:25:46.750838 725 consensus_queue.cc:237] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:46.751518 720 ts_tablet_manager.cc:1398] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20240502 21:25:46.752825 721 tablet_bootstrap.cc:492] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9: No bootstrap required, opened a new log
I20240502 21:25:46.753000 721 ts_tablet_manager.cc:1367] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240502 21:25:46.753842 721 raft_consensus.cc:348] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:46.754076 721 raft_consensus.cc:374] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:46.754168 721 raft_consensus.cc:729] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb8caceef23e475f8304b1ec1175e4b9, State: Initialized, Role: FOLLOWER
I20240502 21:25:46.754441 721 consensus_queue.cc:260] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:46.754575 724 raft_consensus.cc:686] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Becoming Leader. State: Replica: f6f8b5e2067c4c23b0c89a8c471626f5, State: Running, Role: LEADER
I20240502 21:25:46.754462 31978 catalog_manager.cc:5484] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "cb8caceef23e475f8304b1ec1175e4b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } }
I20240502 21:25:46.754673 721 raft_consensus.cc:388] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:46.754786 721 raft_consensus.cc:482] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:46.754918 721 raft_consensus.cc:3037] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.754878 724 consensus_queue.cc:237] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:46.760071 721 raft_consensus.cc:504] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:46.760382 721 leader_election.cc:302] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb8caceef23e475f8304b1ec1175e4b9; no voters:
I20240502 21:25:46.760676 721 leader_election.cc:288] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:46.760812 727 raft_consensus.cc:2781] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:46.761528 721 ts_tablet_manager.cc:1398] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20240502 21:25:46.762027 727 raft_consensus.cc:686] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Becoming Leader. State: Replica: cb8caceef23e475f8304b1ec1175e4b9, State: Running, Role: LEADER
I20240502 21:25:46.762396 727 consensus_queue.cc:237] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:46.765651 31980 catalog_manager.cc:5484] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } }
I20240502 21:25:46.766443 31978 catalog_manager.cc:5484] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "cb8caceef23e475f8304b1ec1175e4b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } }
W20240502 21:25:47.100239 737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:47.100590 737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:47.107880 737 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 103.041 ms
time per row: 0.0515207 ms
I20240502 21:25:47.357053 31978 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51326:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:25:47.365697 32241 tablet_service.cc:1467] Processing CreateTablet for tablet cd16f492f22c478a946d1da05f6a91b4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0e0c0c30a67c4461bafc601a7c3a4676]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:47.366183 32241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd16f492f22c478a946d1da05f6a91b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:47.366153 32107 tablet_service.cc:1467] Processing CreateTablet for tablet 60735476392d4c1492e9227c7f72e18a (DEFAULT_TABLE table=rebalance_test_table_4 [id=0e0c0c30a67c4461bafc601a7c3a4676]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:47.366533 32107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60735476392d4c1492e9227c7f72e18a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:47.365697 32242 tablet_service.cc:1467] Processing CreateTablet for tablet 80471cfd4a944776baf05c4179f64a10 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0e0c0c30a67c4461bafc601a7c3a4676]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:47.368031 32242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80471cfd4a944776baf05c4179f64a10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:47.371188 779 tablet_bootstrap.cc:492] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5: Bootstrap starting.
I20240502 21:25:47.373006 779 tablet_bootstrap.cc:654] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:47.374874 779 tablet_bootstrap.cc:492] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5: No bootstrap required, opened a new log
I20240502 21:25:47.375025 779 ts_tablet_manager.cc:1367] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:25:47.375885 781 tablet_bootstrap.cc:492] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9: Bootstrap starting.
I20240502 21:25:47.375859 779 raft_consensus.cc:348] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:47.376056 779 raft_consensus.cc:374] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:47.376163 779 raft_consensus.cc:729] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f8b5e2067c4c23b0c89a8c471626f5, State: Initialized, Role: FOLLOWER
I20240502 21:25:47.376402 779 consensus_queue.cc:260] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:47.376608 779 raft_consensus.cc:388] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:47.376705 779 raft_consensus.cc:482] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:47.376827 779 raft_consensus.cc:3037] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:47.377632 781 tablet_bootstrap.cc:654] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:47.379315 779 raft_consensus.cc:504] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:47.379524 781 tablet_bootstrap.cc:492] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9: No bootstrap required, opened a new log
I20240502 21:25:47.379591 779 leader_election.cc:302] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6f8b5e2067c4c23b0c89a8c471626f5; no voters:
I20240502 21:25:47.379699 781 ts_tablet_manager.cc:1367] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20240502 21:25:47.379838 779 leader_election.cc:288] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:47.379921 724 raft_consensus.cc:2781] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:47.380673 779 ts_tablet_manager.cc:1398] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20240502 21:25:47.380924 779 tablet_bootstrap.cc:492] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5: Bootstrap starting.
I20240502 21:25:47.380522 781 raft_consensus.cc:348] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:47.381114 781 raft_consensus.cc:374] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:47.381214 781 raft_consensus.cc:729] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb8caceef23e475f8304b1ec1175e4b9, State: Initialized, Role: FOLLOWER
I20240502 21:25:47.381537 781 consensus_queue.cc:260] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:47.381880 781 raft_consensus.cc:388] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:47.382010 781 raft_consensus.cc:482] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:47.382153 781 raft_consensus.cc:3037] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:47.382721 779 tablet_bootstrap.cc:654] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:47.384831 781 raft_consensus.cc:504] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:47.385154 779 tablet_bootstrap.cc:492] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5: No bootstrap required, opened a new log
I20240502 21:25:47.385175 781 leader_election.cc:302] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb8caceef23e475f8304b1ec1175e4b9; no voters:
I20240502 21:25:47.385324 779 ts_tablet_manager.cc:1367] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 21:25:47.385377 781 leader_election.cc:288] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:47.385491 727 raft_consensus.cc:2781] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:47.386090 781 ts_tablet_manager.cc:1398] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20240502 21:25:47.386171 779 raft_consensus.cc:348] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:47.386422 779 raft_consensus.cc:374] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:47.386574 779 raft_consensus.cc:729] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f8b5e2067c4c23b0c89a8c471626f5, State: Initialized, Role: FOLLOWER
I20240502 21:25:47.386569 727 raft_consensus.cc:686] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Becoming Leader. State: Replica: cb8caceef23e475f8304b1ec1175e4b9, State: Running, Role: LEADER
I20240502 21:25:47.386876 779 consensus_queue.cc:260] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:47.386863 727 consensus_queue.cc:237] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:47.387025 779 raft_consensus.cc:388] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:47.387115 779 raft_consensus.cc:482] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:47.387183 779 raft_consensus.cc:3037] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:47.389678 779 raft_consensus.cc:504] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:47.389941 779 leader_election.cc:302] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6f8b5e2067c4c23b0c89a8c471626f5; no voters:
I20240502 21:25:47.390147 779 leader_election.cc:288] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:47.390030 31978 catalog_manager.cc:5484] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "cb8caceef23e475f8304b1ec1175e4b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } }
I20240502 21:25:47.390269 783 raft_consensus.cc:2781] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:47.390611 779 ts_tablet_manager.cc:1398] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 21:25:47.396760 724 raft_consensus.cc:686] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Becoming Leader. State: Replica: f6f8b5e2067c4c23b0c89a8c471626f5, State: Running, Role: LEADER
I20240502 21:25:47.397110 724 consensus_queue.cc:237] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:47.399816 783 raft_consensus.cc:686] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Becoming Leader. State: Replica: f6f8b5e2067c4c23b0c89a8c471626f5, State: Running, Role: LEADER
I20240502 21:25:47.400038 783 consensus_queue.cc:237] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:47.403111 31978 catalog_manager.cc:5484] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } }
I20240502 21:25:47.403477 31978 catalog_manager.cc:5484] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } }
W20240502 21:25:47.733989 791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:47.734299 791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:47.741631 791 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 110.225 ms
time per row: 0.0551123 ms
I20240502 21:25:47.996181 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.67:45329
--local_ip_for_outbound_sockets=127.29.82.67
--tserver_master_addrs=127.29.82.126:44793
--webserver_port=41011
--webserver_interface=127.29.82.67
--builtin_ntp_servers=127.29.82.84:43193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 21:25:48.323653 828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:48.323964 828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:48.324085 828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:48.331774 828 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:25:48.331903 828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:48.332062 828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.67
I20240502 21:25:48.341231 828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:43193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.67:45329
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.29.82.67
--webserver_port=41011
--tserver_master_addrs=127.29.82.126:44793
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.67
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:48.342583 828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:48.344373 828 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:48.357729 833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:48.358240 834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:48.359483 836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:48.360738 828 server_base.cc:1034] running on GCE node
I20240502 21:25:48.361330 828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:48.362334 828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:48.363552 828 hybrid_clock.cc:648] HybridClock initialized: now 1714685148363510 us; error 39 us; skew 500 ppm
I20240502 21:25:48.363912 828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:48.366101 828 webserver.cc:460] Webserver started at http://127.29.82.67:41011/ using document root <none> and password file <none>
I20240502 21:25:48.366822 828 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:48.366955 828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:48.372179 828 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20240502 21:25:48.375118 842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:48.376416 828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:25:48.376575 828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b399bf9ec0c544749c7eead22ddfedf6"
format_stamp: "Formatted at 2024-05-02 21:25:37 on dist-test-slave-xzsm"
I20240502 21:25:48.377398 828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:48.393996 828 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:48.395123 828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:48.395478 828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:48.397087 828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:48.399291 828 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:48.399418 828 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:48.399559 828 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:48.399645 828 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:48.435088 828 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.67:45329
I20240502 21:25:48.435125 956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.67:45329 every 8 connection(s)
I20240502 21:25:48.436702 828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:25:48.445866 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 828
I20240502 21:25:48.446455 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.68:33951
--local_ip_for_outbound_sockets=127.29.82.68
--tserver_master_addrs=127.29.82.126:44793
--webserver_port=45327
--webserver_interface=127.29.82.68
--builtin_ntp_servers=127.29.82.84:43193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:25:48.449395 957 heartbeater.cc:347] Connected to a master server at 127.29.82.126:44793
I20240502 21:25:48.449692 957 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:48.450403 957 heartbeater.cc:511] Master 127.29.82.126:44793 requested a full tablet report, sending...
I20240502 21:25:48.452016 31971 ts_manager.cc:194] Re-registered known tserver with Master: b399bf9ec0c544749c7eead22ddfedf6 (127.29.82.67:45329)
I20240502 21:25:48.453227 31971 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.67:58615
W20240502 21:25:48.781847 962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:48.782158 962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:48.782286 962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:48.789702 962 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:25:48.789819 962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:48.789935 962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.68
I20240502 21:25:48.798641 962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:43193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.68:33951
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.29.82.68
--webserver_port=45327
--tserver_master_addrs=127.29.82.126:44793
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.68
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:48.799933 962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:48.801604 962 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:48.813606 967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:48.814677 968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:48.815655 970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:48.816963 962 server_base.cc:1034] running on GCE node
I20240502 21:25:48.817495 962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:48.818400 962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:48.819610 962 hybrid_clock.cc:648] HybridClock initialized: now 1714685148819540 us; error 47 us; skew 500 ppm
I20240502 21:25:48.819903 962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:48.822067 962 webserver.cc:460] Webserver started at http://127.29.82.68:45327/ using document root <none> and password file <none>
I20240502 21:25:48.822743 962 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:48.822861 962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:48.827944 962 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20240502 21:25:48.830502 976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:48.831784 962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:25:48.831934 962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "f58850eb82214ad39880e45777bf51b4"
format_stamp: "Formatted at 2024-05-02 21:25:37 on dist-test-slave-xzsm"
I20240502 21:25:48.832731 962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:48.856787 962 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:48.857638 962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:48.857940 962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:48.858911 962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:48.860441 962 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:48.860520 962 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:48.860673 962 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:48.860728 962 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:48.893429 962 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.68:33951
I20240502 21:25:48.893474 1090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.68:33951 every 8 connection(s)
I20240502 21:25:48.894910 962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:25:48.896399 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 962
I20240502 21:25:48.897010 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.69:36679
--local_ip_for_outbound_sockets=127.29.82.69
--tserver_master_addrs=127.29.82.126:44793
--webserver_port=33659
--webserver_interface=127.29.82.69
--builtin_ntp_servers=127.29.82.84:43193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:25:48.908463 1091 heartbeater.cc:347] Connected to a master server at 127.29.82.126:44793
I20240502 21:25:48.908802 1091 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:48.909682 1091 heartbeater.cc:511] Master 127.29.82.126:44793 requested a full tablet report, sending...
I20240502 21:25:48.911375 31971 ts_manager.cc:194] Re-registered known tserver with Master: f58850eb82214ad39880e45777bf51b4 (127.29.82.68:33951)
I20240502 21:25:48.912643 31971 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.68:59313
W20240502 21:25:49.229790 1094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:49.230109 1094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:49.230234 1094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:49.238291 1094 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:25:49.238422 1094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:49.238601 1094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.69
I20240502 21:25:49.247538 1094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:43193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.69:36679
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.29.82.69
--webserver_port=33659
--tserver_master_addrs=127.29.82.126:44793
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.69
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:49.248886 1094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:49.250627 1094 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:49.263209 1102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:49.263867 1099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:49.264912 1100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:49.266145 1094 server_base.cc:1034] running on GCE node
I20240502 21:25:49.266773 1094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:49.267735 1094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:49.268936 1094 hybrid_clock.cc:648] HybridClock initialized: now 1714685149268885 us; error 37 us; skew 500 ppm
I20240502 21:25:49.269253 1094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:49.271483 1094 webserver.cc:460] Webserver started at http://127.29.82.69:33659/ using document root <none> and password file <none>
I20240502 21:25:49.272279 1094 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:49.272405 1094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:49.277731 1094 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 21:25:49.280632 1109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:49.282075 1094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 21:25:49.282337 1094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "21e73dc1a0eb486db9717fa320347e1c"
format_stamp: "Formatted at 2024-05-02 21:25:38 on dist-test-slave-xzsm"
I20240502 21:25:49.283210 1094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:49.309288 1094 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:49.310164 1094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:49.310459 1094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:49.311475 1094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:49.313064 1094 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:49.313148 1094 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:49.313266 1094 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:49.313318 1094 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:49.346500 1222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.69:36679 every 8 connection(s)
I20240502 21:25:49.346468 1094 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.69:36679
I20240502 21:25:49.348156 1094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:25:49.357802 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 1094
I20240502 21:25:49.358510 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.70:45931
--local_ip_for_outbound_sockets=127.29.82.70
--tserver_master_addrs=127.29.82.126:44793
--webserver_port=37091
--webserver_interface=127.29.82.70
--builtin_ntp_servers=127.29.82.84:43193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:25:49.361157 1223 heartbeater.cc:347] Connected to a master server at 127.29.82.126:44793
I20240502 21:25:49.361443 1223 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:49.362121 1223 heartbeater.cc:511] Master 127.29.82.126:44793 requested a full tablet report, sending...
I20240502 21:25:49.363659 31971 ts_manager.cc:194] Re-registered known tserver with Master: 21e73dc1a0eb486db9717fa320347e1c (127.29.82.69:36679)
I20240502 21:25:49.364848 31971 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.69:34187
I20240502 21:25:49.455785 957 heartbeater.cc:503] Master 127.29.82.126:44793 was elected leader, sending a full tablet report...
W20240502 21:25:49.689018 1226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:49.689342 1226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:49.689491 1226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:49.699996 1226 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:25:49.700107 1226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:49.700263 1226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.70
I20240502 21:25:49.709326 1226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:43193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.70:45931
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.29.82.70
--webserver_port=37091
--tserver_master_addrs=127.29.82.126:44793
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.70
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:49.710733 1226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:49.712539 1226 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:49.726019 1232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:49.726475 1231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:49.727346 1234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:49.728690 1226 server_base.cc:1034] running on GCE node
I20240502 21:25:49.729271 1226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:49.730206 1226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:49.731397 1226 hybrid_clock.cc:648] HybridClock initialized: now 1714685149731341 us; error 45 us; skew 500 ppm
I20240502 21:25:49.731755 1226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:49.733851 1226 webserver.cc:460] Webserver started at http://127.29.82.70:37091/ using document root <none> and password file <none>
I20240502 21:25:49.734534 1226 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:49.734660 1226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:49.739871 1226 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:25:49.742502 1240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:49.743798 1226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:25:49.743948 1226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "e705f0dda7904364a7453ee6294b3601"
format_stamp: "Formatted at 2024-05-02 21:25:38 on dist-test-slave-xzsm"
I20240502 21:25:49.744740 1226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:49.772867 1226 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:49.773679 1226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:49.773972 1226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:49.774938 1226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:49.776500 1226 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:49.776578 1226 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:49.776706 1226 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:49.776769 1226 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:49.809582 1226 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.70:45931
I20240502 21:25:49.809628 1353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.70:45931 every 8 connection(s)
I20240502 21:25:49.811004 1226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 21:25:49.818193 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 1226
I20240502 21:25:49.819005 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.71:39493
--local_ip_for_outbound_sockets=127.29.82.71
--tserver_master_addrs=127.29.82.126:44793
--webserver_port=44147
--webserver_interface=127.29.82.71
--builtin_ntp_servers=127.29.82.84:43193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:25:49.823958 1354 heartbeater.cc:347] Connected to a master server at 127.29.82.126:44793
I20240502 21:25:49.824236 1354 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:49.824903 1354 heartbeater.cc:511] Master 127.29.82.126:44793 requested a full tablet report, sending...
I20240502 21:25:49.826210 31971 ts_manager.cc:194] Re-registered known tserver with Master: e705f0dda7904364a7453ee6294b3601 (127.29.82.70:45931)
I20240502 21:25:49.827059 31971 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.70:33425
I20240502 21:25:49.915593 1091 heartbeater.cc:503] Master 127.29.82.126:44793 was elected leader, sending a full tablet report...
W20240502 21:25:50.157243 1357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:50.157523 1357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:50.157610 1357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:50.164913 1357 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:25:50.165032 1357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:50.165148 1357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.71
I20240502 21:25:50.174063 1357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:43193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.71:39493
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.29.82.71
--webserver_port=44147
--tserver_master_addrs=127.29.82.126:44793
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.71
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:50.175374 1357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:50.177376 1357 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:50.190619 1363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:50.191530 1365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:50.192221 1362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:50.193351 1357 server_base.cc:1034] running on GCE node
I20240502 21:25:50.193948 1357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:50.194898 1357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:50.196094 1357 hybrid_clock.cc:648] HybridClock initialized: now 1714685150196051 us; error 37 us; skew 500 ppm
I20240502 21:25:50.196413 1357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:50.198637 1357 webserver.cc:460] Webserver started at http://127.29.82.71:44147/ using document root <none> and password file <none>
I20240502 21:25:50.199363 1357 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:50.199497 1357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:50.204761 1357 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20240502 21:25:50.207384 1371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:50.208624 1357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 21:25:50.208798 1357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "98ac0bd5300e42c38bee7aae9634c7c9"
format_stamp: "Formatted at 2024-05-02 21:25:39 on dist-test-slave-xzsm"
I20240502 21:25:50.209695 1357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:50.242991 1357 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:50.243872 1357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:50.244163 1357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:50.245083 1357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:50.246543 1357 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:50.246627 1357 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:50.246788 1357 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:50.246845 1357 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:50.278379 1357 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.71:39493
I20240502 21:25:50.278472 1484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.71:39493 every 8 connection(s)
I20240502 21:25:50.279942 1357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 21:25:50.288780 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 1357
I20240502 21:25:50.292999 1485 heartbeater.cc:347] Connected to a master server at 127.29.82.126:44793
I20240502 21:25:50.293298 1485 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:50.293983 1485 heartbeater.cc:511] Master 127.29.82.126:44793 requested a full tablet report, sending...
I20240502 21:25:50.295380 31971 ts_manager.cc:194] Re-registered known tserver with Master: 98ac0bd5300e42c38bee7aae9634c7c9 (127.29.82.71:39493)
I20240502 21:25:50.296171 31971 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.71:33949
I20240502 21:25:50.367414 1223 heartbeater.cc:503] Master 127.29.82.126:44793 was elected leader, sending a full tablet report...
I20240502 21:25:50.771380 1420 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:50.795514 1157 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:50.818918 32242 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:25:50.829310 1354 heartbeater.cc:503] Master 127.29.82.126:44793 was elected leader, sending a full tablet report...
I20240502 21:25:50.829645 1024 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:50.831836 1288 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:50.832140 32107 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:25:50.841989 890 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:51.055295 1157 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:51.064777 1420 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:51.068923 890 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:51.070806 32107 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:25:51.088747 1024 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:51.107998 1288 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:51.116907 32242 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:25:51.204473 32127 consensus_queue.cc:237] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "98ac0bd5300e42c38bee7aae9634c7c9" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 39493 } }
W20240502 21:25:51.214035 32059 consensus_peers.cc:485] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9 -> Peer 98ac0bd5300e42c38bee7aae9634c7c9 (127.29.82.71:39493): Couldn't send request to peer 98ac0bd5300e42c38bee7aae9634c7c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4919235c52834df5b3924d93391f171a. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:51.298363 1485 heartbeater.cc:503] Master 127.29.82.126:44793 was elected leader, sending a full tablet report...
I20240502 21:25:51.707024 1587 ts_tablet_manager.cc:902] T 4919235c52834df5b3924d93391f171a P 98ac0bd5300e42c38bee7aae9634c7c9: Initiating tablet copy from peer cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65:37947)
I20240502 21:25:51.708834 1587 tablet_copy_client.cc:308] T 4919235c52834df5b3924d93391f171a P 98ac0bd5300e42c38bee7aae9634c7c9: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.65:37947
I20240502 21:25:51.715889 32147 tablet_copy_service.cc:140] P cb8caceef23e475f8304b1ec1175e4b9: Received BeginTabletCopySession request for tablet 4919235c52834df5b3924d93391f171a from peer 98ac0bd5300e42c38bee7aae9634c7c9 ({username='slave'} at 127.29.82.71:57261)
I20240502 21:25:51.716130 32147 tablet_copy_service.cc:161] P cb8caceef23e475f8304b1ec1175e4b9: Beginning new tablet copy session on tablet 4919235c52834df5b3924d93391f171a from peer 98ac0bd5300e42c38bee7aae9634c7c9 at {username='slave'} at 127.29.82.71:57261: session id = 98ac0bd5300e42c38bee7aae9634c7c9-4919235c52834df5b3924d93391f171a
I20240502 21:25:51.718760 32147 tablet_copy_source_session.cc:205] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:25:51.723507 1587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4919235c52834df5b3924d93391f171a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:51.733578 1587 tablet_copy_client.cc:783] T 4919235c52834df5b3924d93391f171a P 98ac0bd5300e42c38bee7aae9634c7c9: tablet copy: Starting download of 0 data blocks...
I20240502 21:25:51.733898 1587 tablet_copy_client.cc:647] T 4919235c52834df5b3924d93391f171a P 98ac0bd5300e42c38bee7aae9634c7c9: tablet copy: Starting download of 1 WAL segments...
I20240502 21:25:51.736202 1587 tablet_copy_client.cc:523] T 4919235c52834df5b3924d93391f171a P 98ac0bd5300e42c38bee7aae9634c7c9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:25:51.739235 1587 tablet_bootstrap.cc:492] T 4919235c52834df5b3924d93391f171a P 98ac0bd5300e42c38bee7aae9634c7c9: Bootstrap starting.
I20240502 21:25:51.747990 1587 log.cc:824] T 4919235c52834df5b3924d93391f171a P 98ac0bd5300e42c38bee7aae9634c7c9: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:51.779002 1587 tablet_bootstrap.cc:492] T 4919235c52834df5b3924d93391f171a P 98ac0bd5300e42c38bee7aae9634c7c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 21:25:51.779546 1587 tablet_bootstrap.cc:492] T 4919235c52834df5b3924d93391f171a P 98ac0bd5300e42c38bee7aae9634c7c9: Bootstrap complete.
I20240502 21:25:51.780083 1587 ts_tablet_manager.cc:1367] T 4919235c52834df5b3924d93391f171a P 98ac0bd5300e42c38bee7aae9634c7c9: Time spent bootstrapping tablet: real 0.041s user 0.039s sys 0.004s
I20240502 21:25:51.784785 1587 raft_consensus.cc:348] T 4919235c52834df5b3924d93391f171a P 98ac0bd5300e42c38bee7aae9634c7c9 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:51.785705 1587 raft_consensus.cc:729] T 4919235c52834df5b3924d93391f171a P 98ac0bd5300e42c38bee7aae9634c7c9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98ac0bd5300e42c38bee7aae9634c7c9, State: Initialized, Role: FOLLOWER
I20240502 21:25:51.786142 1587 consensus_queue.cc:260] T 4919235c52834df5b3924d93391f171a P 98ac0bd5300e42c38bee7aae9634c7c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "98ac0bd5300e42c38bee7aae9634c7c9" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 39493 } }
I20240502 21:25:51.787851 1587 ts_tablet_manager.cc:1398] T 4919235c52834df5b3924d93391f171a P 98ac0bd5300e42c38bee7aae9634c7c9: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240502 21:25:51.789175 32147 tablet_copy_service.cc:342] P cb8caceef23e475f8304b1ec1175e4b9: Request end of tablet copy session 98ac0bd5300e42c38bee7aae9634c7c9-4919235c52834df5b3924d93391f171a received from {username='slave'} at 127.29.82.71:57261
I20240502 21:25:51.789350 32147 tablet_copy_service.cc:433] P cb8caceef23e475f8304b1ec1175e4b9: ending tablet copy session 98ac0bd5300e42c38bee7aae9634c7c9-4919235c52834df5b3924d93391f171a on tablet 4919235c52834df5b3924d93391f171a with peer 98ac0bd5300e42c38bee7aae9634c7c9
I20240502 21:25:52.251736 1440 raft_consensus.cc:1203] T 4919235c52834df5b3924d93391f171a P 98ac0bd5300e42c38bee7aae9634c7c9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:25:52.253603 1597 raft_consensus.cc:2932] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 98ac0bd5300e42c38bee7aae9634c7c9 (127.29.82.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "98ac0bd5300e42c38bee7aae9634c7c9" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 39493 } } }
I20240502 21:25:52.254207 1440 raft_consensus.cc:2932] T 4919235c52834df5b3924d93391f171a P 98ac0bd5300e42c38bee7aae9634c7c9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 98ac0bd5300e42c38bee7aae9634c7c9 (127.29.82.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "98ac0bd5300e42c38bee7aae9634c7c9" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 39493 } } }
I20240502 21:25:52.257186 1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:41468 (request call id 0) took 1053 ms. Trace:
I20240502 21:25:52.257539 1597 rpcz_store.cc:276] 0502 21:25:51.203359 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:25:51.203425 (+ 66us) service_pool.cc:225] Handling call
0502 21:25:51.205709 (+ 2284us) log.cc:842] Serialized 257 byte log entry
0502 21:25:52.257123 (+1051414us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":73,"raft.queue_time_us":390,"raft.run_cpu_time_us":520,"raft.run_wall_time_us":520,"spinlock_wait_cycles":97664,"thread_start_us":500,"threads_started":2,"wal-append.queue_time_us":329,"wal-append.run_cpu_time_us":592,"wal-append.run_wall_time_us":1000694}
I20240502 21:25:52.259220 31978 catalog_manager.cc:5484] T 4919235c52834df5b3924d93391f171a P 98ac0bd5300e42c38bee7aae9634c7c9 reported cstate change: config changed from index -1 to 4, VOTER 98ac0bd5300e42c38bee7aae9634c7c9 (127.29.82.71) added. New cstate: current_term: 1 leader_uuid: "cb8caceef23e475f8304b1ec1175e4b9" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "98ac0bd5300e42c38bee7aae9634c7c9" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 39493 } } }
I20240502 21:25:52.302080 32263 consensus_queue.cc:237] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "f58850eb82214ad39880e45777bf51b4" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 33951 } }
W20240502 21:25:52.311769 32194 consensus_peers.cc:485] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5 -> Peer f58850eb82214ad39880e45777bf51b4 (127.29.82.68:33951): Couldn't send request to peer f58850eb82214ad39880e45777bf51b4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80471cfd4a944776baf05c4179f64a10. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:52.759961 1625 ts_tablet_manager.cc:902] T 80471cfd4a944776baf05c4179f64a10 P f58850eb82214ad39880e45777bf51b4: Initiating tablet copy from peer f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66:39503)
I20240502 21:25:52.761685 1625 tablet_copy_client.cc:308] T 80471cfd4a944776baf05c4179f64a10 P f58850eb82214ad39880e45777bf51b4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.66:39503
I20240502 21:25:52.768115 32283 tablet_copy_service.cc:140] P f6f8b5e2067c4c23b0c89a8c471626f5: Received BeginTabletCopySession request for tablet 80471cfd4a944776baf05c4179f64a10 from peer f58850eb82214ad39880e45777bf51b4 ({username='slave'} at 127.29.82.68:59939)
I20240502 21:25:52.768332 32283 tablet_copy_service.cc:161] P f6f8b5e2067c4c23b0c89a8c471626f5: Beginning new tablet copy session on tablet 80471cfd4a944776baf05c4179f64a10 from peer f58850eb82214ad39880e45777bf51b4 at {username='slave'} at 127.29.82.68:59939: session id = f58850eb82214ad39880e45777bf51b4-80471cfd4a944776baf05c4179f64a10
I20240502 21:25:52.770910 32283 tablet_copy_source_session.cc:205] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:25:52.775444 1625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80471cfd4a944776baf05c4179f64a10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:52.785697 1625 tablet_copy_client.cc:783] T 80471cfd4a944776baf05c4179f64a10 P f58850eb82214ad39880e45777bf51b4: tablet copy: Starting download of 0 data blocks...
I20240502 21:25:52.786031 1625 tablet_copy_client.cc:647] T 80471cfd4a944776baf05c4179f64a10 P f58850eb82214ad39880e45777bf51b4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:25:52.788581 1625 tablet_copy_client.cc:523] T 80471cfd4a944776baf05c4179f64a10 P f58850eb82214ad39880e45777bf51b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:25:52.792043 1625 tablet_bootstrap.cc:492] T 80471cfd4a944776baf05c4179f64a10 P f58850eb82214ad39880e45777bf51b4: Bootstrap starting.
I20240502 21:25:52.801095 1625 log.cc:824] T 80471cfd4a944776baf05c4179f64a10 P f58850eb82214ad39880e45777bf51b4: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:52.831228 1625 tablet_bootstrap.cc:492] T 80471cfd4a944776baf05c4179f64a10 P f58850eb82214ad39880e45777bf51b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 21:25:52.831873 1625 tablet_bootstrap.cc:492] T 80471cfd4a944776baf05c4179f64a10 P f58850eb82214ad39880e45777bf51b4: Bootstrap complete.
I20240502 21:25:52.832334 1625 ts_tablet_manager.cc:1367] T 80471cfd4a944776baf05c4179f64a10 P f58850eb82214ad39880e45777bf51b4: Time spent bootstrapping tablet: real 0.040s user 0.039s sys 0.000s
I20240502 21:25:52.837397 1625 raft_consensus.cc:348] T 80471cfd4a944776baf05c4179f64a10 P f58850eb82214ad39880e45777bf51b4 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:52.838307 1625 raft_consensus.cc:729] T 80471cfd4a944776baf05c4179f64a10 P f58850eb82214ad39880e45777bf51b4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58850eb82214ad39880e45777bf51b4, State: Initialized, Role: FOLLOWER
I20240502 21:25:52.838765 1625 consensus_queue.cc:260] T 80471cfd4a944776baf05c4179f64a10 P f58850eb82214ad39880e45777bf51b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "f58850eb82214ad39880e45777bf51b4" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 33951 } }
I20240502 21:25:52.840209 1625 ts_tablet_manager.cc:1398] T 80471cfd4a944776baf05c4179f64a10 P f58850eb82214ad39880e45777bf51b4: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240502 21:25:52.841213 32283 tablet_copy_service.cc:342] P f6f8b5e2067c4c23b0c89a8c471626f5: Request end of tablet copy session f58850eb82214ad39880e45777bf51b4-80471cfd4a944776baf05c4179f64a10 received from {username='slave'} at 127.29.82.68:59939
I20240502 21:25:52.841385 32283 tablet_copy_service.cc:433] P f6f8b5e2067c4c23b0c89a8c471626f5: ending tablet copy session f58850eb82214ad39880e45777bf51b4-80471cfd4a944776baf05c4179f64a10 on tablet 80471cfd4a944776baf05c4179f64a10 with peer f58850eb82214ad39880e45777bf51b4
I20240502 21:25:53.336016 1046 raft_consensus.cc:1203] T 80471cfd4a944776baf05c4179f64a10 P f58850eb82214ad39880e45777bf51b4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:25:53.338153 1635 raft_consensus.cc:2932] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f58850eb82214ad39880e45777bf51b4 (127.29.82.68) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "f58850eb82214ad39880e45777bf51b4" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 33951 } } }
I20240502 21:25:53.338763 1046 raft_consensus.cc:2932] T 80471cfd4a944776baf05c4179f64a10 P f58850eb82214ad39880e45777bf51b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f58850eb82214ad39880e45777bf51b4 (127.29.82.68) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "f58850eb82214ad39880e45777bf51b4" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 33951 } } }
I20240502 21:25:53.341650 1635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:46414 (request call id 0) took 1040 ms. Trace:
I20240502 21:25:53.341818 1635 rpcz_store.cc:276] 0502 21:25:52.300958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:25:52.301034 (+ 76us) service_pool.cc:225] Handling call
0502 21:25:52.303350 (+ 2316us) log.cc:842] Serialized 257 byte log entry
0502 21:25:53.341594 (+1038244us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":72,"raft.queue_time_us":387,"raft.run_cpu_time_us":508,"raft.run_wall_time_us":508,"spinlock_wait_cycles":45312,"thread_start_us":548,"threads_started":2,"wal-append.queue_time_us":353,"wal-append.run_cpu_time_us":557,"wal-append.run_wall_time_us":1000629}
I20240502 21:25:53.344215 31980 catalog_manager.cc:5484] T 80471cfd4a944776baf05c4179f64a10 P f58850eb82214ad39880e45777bf51b4 reported cstate change: config changed from index -1 to 4, VOTER f58850eb82214ad39880e45777bf51b4 (127.29.82.68) added. New cstate: current_term: 1 leader_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "f58850eb82214ad39880e45777bf51b4" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 33951 } } }
I20240502 21:25:53.379036 32127 consensus_queue.cc:237] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "b399bf9ec0c544749c7eead22ddfedf6" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 45329 } }
W20240502 21:25:53.387168 32061 consensus_peers.cc:485] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9 -> Peer b399bf9ec0c544749c7eead22ddfedf6 (127.29.82.67:45329): Couldn't send request to peer b399bf9ec0c544749c7eead22ddfedf6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 699c3a3f730248bdacf122126883f2f4. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:53.966038 1662 ts_tablet_manager.cc:902] T 699c3a3f730248bdacf122126883f2f4 P b399bf9ec0c544749c7eead22ddfedf6: Initiating tablet copy from peer cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65:37947)
I20240502 21:25:53.967808 1662 tablet_copy_client.cc:308] T 699c3a3f730248bdacf122126883f2f4 P b399bf9ec0c544749c7eead22ddfedf6: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.65:37947
I20240502 21:25:53.974402 32147 tablet_copy_service.cc:140] P cb8caceef23e475f8304b1ec1175e4b9: Received BeginTabletCopySession request for tablet 699c3a3f730248bdacf122126883f2f4 from peer b399bf9ec0c544749c7eead22ddfedf6 ({username='slave'} at 127.29.82.67:56567)
I20240502 21:25:53.974566 32147 tablet_copy_service.cc:161] P cb8caceef23e475f8304b1ec1175e4b9: Beginning new tablet copy session on tablet 699c3a3f730248bdacf122126883f2f4 from peer b399bf9ec0c544749c7eead22ddfedf6 at {username='slave'} at 127.29.82.67:56567: session id = b399bf9ec0c544749c7eead22ddfedf6-699c3a3f730248bdacf122126883f2f4
I20240502 21:25:53.976611 32147 tablet_copy_source_session.cc:205] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:25:53.981016 1662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 699c3a3f730248bdacf122126883f2f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:53.991410 1662 tablet_copy_client.cc:783] T 699c3a3f730248bdacf122126883f2f4 P b399bf9ec0c544749c7eead22ddfedf6: tablet copy: Starting download of 0 data blocks...
I20240502 21:25:53.991725 1662 tablet_copy_client.cc:647] T 699c3a3f730248bdacf122126883f2f4 P b399bf9ec0c544749c7eead22ddfedf6: tablet copy: Starting download of 1 WAL segments...
I20240502 21:25:53.993881 1662 tablet_copy_client.cc:523] T 699c3a3f730248bdacf122126883f2f4 P b399bf9ec0c544749c7eead22ddfedf6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:25:53.996937 1662 tablet_bootstrap.cc:492] T 699c3a3f730248bdacf122126883f2f4 P b399bf9ec0c544749c7eead22ddfedf6: Bootstrap starting.
I20240502 21:25:54.005954 1662 log.cc:824] T 699c3a3f730248bdacf122126883f2f4 P b399bf9ec0c544749c7eead22ddfedf6: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:54.036937 1662 tablet_bootstrap.cc:492] T 699c3a3f730248bdacf122126883f2f4 P b399bf9ec0c544749c7eead22ddfedf6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 21:25:54.037539 1662 tablet_bootstrap.cc:492] T 699c3a3f730248bdacf122126883f2f4 P b399bf9ec0c544749c7eead22ddfedf6: Bootstrap complete.
I20240502 21:25:54.037957 1662 ts_tablet_manager.cc:1367] T 699c3a3f730248bdacf122126883f2f4 P b399bf9ec0c544749c7eead22ddfedf6: Time spent bootstrapping tablet: real 0.041s user 0.039s sys 0.001s
I20240502 21:25:54.042407 1662 raft_consensus.cc:348] T 699c3a3f730248bdacf122126883f2f4 P b399bf9ec0c544749c7eead22ddfedf6 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:54.043208 1662 raft_consensus.cc:729] T 699c3a3f730248bdacf122126883f2f4 P b399bf9ec0c544749c7eead22ddfedf6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b399bf9ec0c544749c7eead22ddfedf6, State: Initialized, Role: FOLLOWER
I20240502 21:25:54.043650 1662 consensus_queue.cc:260] T 699c3a3f730248bdacf122126883f2f4 P b399bf9ec0c544749c7eead22ddfedf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "b399bf9ec0c544749c7eead22ddfedf6" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 45329 } }
I20240502 21:25:54.044965 1662 ts_tablet_manager.cc:1398] T 699c3a3f730248bdacf122126883f2f4 P b399bf9ec0c544749c7eead22ddfedf6: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240502 21:25:54.046084 32147 tablet_copy_service.cc:342] P cb8caceef23e475f8304b1ec1175e4b9: Request end of tablet copy session b399bf9ec0c544749c7eead22ddfedf6-699c3a3f730248bdacf122126883f2f4 received from {username='slave'} at 127.29.82.67:56567
I20240502 21:25:54.046237 32147 tablet_copy_service.cc:433] P cb8caceef23e475f8304b1ec1175e4b9: ending tablet copy session b399bf9ec0c544749c7eead22ddfedf6-699c3a3f730248bdacf122126883f2f4 on tablet 699c3a3f730248bdacf122126883f2f4 with peer b399bf9ec0c544749c7eead22ddfedf6
I20240502 21:25:54.580732 911 raft_consensus.cc:1203] T 699c3a3f730248bdacf122126883f2f4 P b399bf9ec0c544749c7eead22ddfedf6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:25:54.582937 1673 raft_consensus.cc:2932] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b399bf9ec0c544749c7eead22ddfedf6 (127.29.82.67) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "b399bf9ec0c544749c7eead22ddfedf6" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 45329 } } }
I20240502 21:25:54.583706 911 raft_consensus.cc:2932] T 699c3a3f730248bdacf122126883f2f4 P b399bf9ec0c544749c7eead22ddfedf6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b399bf9ec0c544749c7eead22ddfedf6 (127.29.82.67) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "b399bf9ec0c544749c7eead22ddfedf6" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 45329 } } }
I20240502 21:25:54.586169 1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:41480 (request call id 0) took 1207 ms. Trace:
I20240502 21:25:54.586355 1673 rpcz_store.cc:276] 0502 21:25:53.378489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:25:53.378558 (+ 69us) service_pool.cc:225] Handling call
0502 21:25:53.379788 (+ 1230us) log.cc:842] Serialized 257 byte log entry
0502 21:25:54.586098 (+1206310us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":61,"raft.queue_time_us":55,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":219,"thread_start_us":189,"threads_started":1,"wal-append.queue_time_us":266,"wal-append.run_cpu_time_us":579,"wal-append.run_wall_time_us":1000627}
I20240502 21:25:54.589347 31980 catalog_manager.cc:5484] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9 reported cstate change: config changed from index -1 to 4, VOTER b399bf9ec0c544749c7eead22ddfedf6 (127.29.82.67) added. New cstate: current_term: 1 leader_uuid: "cb8caceef23e475f8304b1ec1175e4b9" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "b399bf9ec0c544749c7eead22ddfedf6" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 45329 } } }
I20240502 21:25:54.625779 32127 consensus_queue.cc:237] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "e705f0dda7904364a7453ee6294b3601" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 45931 } }
W20240502 21:25:54.634594 32062 consensus_peers.cc:485] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9 -> Peer e705f0dda7904364a7453ee6294b3601 (127.29.82.70:45931): Couldn't send request to peer e705f0dda7904364a7453ee6294b3601. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d484c2aeb0904564997462a634b541d3. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:55.194141 1702 ts_tablet_manager.cc:902] T d484c2aeb0904564997462a634b541d3 P e705f0dda7904364a7453ee6294b3601: Initiating tablet copy from peer cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65:37947)
I20240502 21:25:55.195961 1702 tablet_copy_client.cc:308] T d484c2aeb0904564997462a634b541d3 P e705f0dda7904364a7453ee6294b3601: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.65:37947
I20240502 21:25:55.202451 32147 tablet_copy_service.cc:140] P cb8caceef23e475f8304b1ec1175e4b9: Received BeginTabletCopySession request for tablet d484c2aeb0904564997462a634b541d3 from peer e705f0dda7904364a7453ee6294b3601 ({username='slave'} at 127.29.82.70:53679)
I20240502 21:25:55.202600 32147 tablet_copy_service.cc:161] P cb8caceef23e475f8304b1ec1175e4b9: Beginning new tablet copy session on tablet d484c2aeb0904564997462a634b541d3 from peer e705f0dda7904364a7453ee6294b3601 at {username='slave'} at 127.29.82.70:53679: session id = e705f0dda7904364a7453ee6294b3601-d484c2aeb0904564997462a634b541d3
I20240502 21:25:55.204669 32147 tablet_copy_source_session.cc:205] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:25:55.208916 1702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d484c2aeb0904564997462a634b541d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:55.219017 1702 tablet_copy_client.cc:783] T d484c2aeb0904564997462a634b541d3 P e705f0dda7904364a7453ee6294b3601: tablet copy: Starting download of 0 data blocks...
I20240502 21:25:55.219316 1702 tablet_copy_client.cc:647] T d484c2aeb0904564997462a634b541d3 P e705f0dda7904364a7453ee6294b3601: tablet copy: Starting download of 1 WAL segments...
I20240502 21:25:55.221349 1702 tablet_copy_client.cc:523] T d484c2aeb0904564997462a634b541d3 P e705f0dda7904364a7453ee6294b3601: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:25:55.224464 1702 tablet_bootstrap.cc:492] T d484c2aeb0904564997462a634b541d3 P e705f0dda7904364a7453ee6294b3601: Bootstrap starting.
I20240502 21:25:55.233193 1702 log.cc:824] T d484c2aeb0904564997462a634b541d3 P e705f0dda7904364a7453ee6294b3601: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:55.262344 1702 tablet_bootstrap.cc:492] T d484c2aeb0904564997462a634b541d3 P e705f0dda7904364a7453ee6294b3601: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 21:25:55.262800 1702 tablet_bootstrap.cc:492] T d484c2aeb0904564997462a634b541d3 P e705f0dda7904364a7453ee6294b3601: Bootstrap complete.
I20240502 21:25:55.263144 1702 ts_tablet_manager.cc:1367] T d484c2aeb0904564997462a634b541d3 P e705f0dda7904364a7453ee6294b3601: Time spent bootstrapping tablet: real 0.039s user 0.036s sys 0.000s
I20240502 21:25:55.267426 1702 raft_consensus.cc:348] T d484c2aeb0904564997462a634b541d3 P e705f0dda7904364a7453ee6294b3601 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:55.268214 1702 raft_consensus.cc:729] T d484c2aeb0904564997462a634b541d3 P e705f0dda7904364a7453ee6294b3601 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e705f0dda7904364a7453ee6294b3601, State: Initialized, Role: FOLLOWER
I20240502 21:25:55.268584 1702 consensus_queue.cc:260] T d484c2aeb0904564997462a634b541d3 P e705f0dda7904364a7453ee6294b3601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "e705f0dda7904364a7453ee6294b3601" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 45931 } }
I20240502 21:25:55.269930 1702 ts_tablet_manager.cc:1398] T d484c2aeb0904564997462a634b541d3 P e705f0dda7904364a7453ee6294b3601: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20240502 21:25:55.271011 32147 tablet_copy_service.cc:342] P cb8caceef23e475f8304b1ec1175e4b9: Request end of tablet copy session e705f0dda7904364a7453ee6294b3601-d484c2aeb0904564997462a634b541d3 received from {username='slave'} at 127.29.82.70:53679
I20240502 21:25:55.271183 32147 tablet_copy_service.cc:433] P cb8caceef23e475f8304b1ec1175e4b9: ending tablet copy session e705f0dda7904364a7453ee6294b3601-d484c2aeb0904564997462a634b541d3 on tablet d484c2aeb0904564997462a634b541d3 with peer e705f0dda7904364a7453ee6294b3601
I20240502 21:25:55.573828 1308 raft_consensus.cc:1203] T d484c2aeb0904564997462a634b541d3 P e705f0dda7904364a7453ee6294b3601 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:25:55.575531 1712 raft_consensus.cc:2932] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e705f0dda7904364a7453ee6294b3601 (127.29.82.70) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "e705f0dda7904364a7453ee6294b3601" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 45931 } } }
I20240502 21:25:55.576417 1308 raft_consensus.cc:2932] T d484c2aeb0904564997462a634b541d3 P e705f0dda7904364a7453ee6294b3601 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e705f0dda7904364a7453ee6294b3601 (127.29.82.70) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "e705f0dda7904364a7453ee6294b3601" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 45931 } } }
I20240502 21:25:55.581513 31978 catalog_manager.cc:5484] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9 reported cstate change: config changed from index -1 to 4, VOTER e705f0dda7904364a7453ee6294b3601 (127.29.82.70) added. New cstate: current_term: 1 leader_uuid: "cb8caceef23e475f8304b1ec1175e4b9" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "e705f0dda7904364a7453ee6294b3601" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 45931 } } }
I20240502 21:25:55.609968 32263 consensus_queue.cc:237] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "21e73dc1a0eb486db9717fa320347e1c" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 36679 } }
W20240502 21:25:55.617971 32194 consensus_peers.cc:485] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5 -> Peer 21e73dc1a0eb486db9717fa320347e1c (127.29.82.69:36679): Couldn't send request to peer 21e73dc1a0eb486db9717fa320347e1c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3843ac68594f43079a1253a877c48065. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:56.101894 1738 ts_tablet_manager.cc:902] T 3843ac68594f43079a1253a877c48065 P 21e73dc1a0eb486db9717fa320347e1c: Initiating tablet copy from peer f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66:39503)
I20240502 21:25:56.103870 1738 tablet_copy_client.cc:308] T 3843ac68594f43079a1253a877c48065 P 21e73dc1a0eb486db9717fa320347e1c: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.66:39503
I20240502 21:25:56.110476 32283 tablet_copy_service.cc:140] P f6f8b5e2067c4c23b0c89a8c471626f5: Received BeginTabletCopySession request for tablet 3843ac68594f43079a1253a877c48065 from peer 21e73dc1a0eb486db9717fa320347e1c ({username='slave'} at 127.29.82.69:44845)
I20240502 21:25:56.110682 32283 tablet_copy_service.cc:161] P f6f8b5e2067c4c23b0c89a8c471626f5: Beginning new tablet copy session on tablet 3843ac68594f43079a1253a877c48065 from peer 21e73dc1a0eb486db9717fa320347e1c at {username='slave'} at 127.29.82.69:44845: session id = 21e73dc1a0eb486db9717fa320347e1c-3843ac68594f43079a1253a877c48065
I20240502 21:25:56.112958 32283 tablet_copy_source_session.cc:205] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:25:56.117565 1738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3843ac68594f43079a1253a877c48065. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:56.128033 1738 tablet_copy_client.cc:783] T 3843ac68594f43079a1253a877c48065 P 21e73dc1a0eb486db9717fa320347e1c: tablet copy: Starting download of 0 data blocks...
I20240502 21:25:56.128329 1738 tablet_copy_client.cc:647] T 3843ac68594f43079a1253a877c48065 P 21e73dc1a0eb486db9717fa320347e1c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:25:56.130554 1738 tablet_copy_client.cc:523] T 3843ac68594f43079a1253a877c48065 P 21e73dc1a0eb486db9717fa320347e1c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:25:56.133735 1738 tablet_bootstrap.cc:492] T 3843ac68594f43079a1253a877c48065 P 21e73dc1a0eb486db9717fa320347e1c: Bootstrap starting.
I20240502 21:25:56.142994 1738 log.cc:824] T 3843ac68594f43079a1253a877c48065 P 21e73dc1a0eb486db9717fa320347e1c: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:56.173274 1738 tablet_bootstrap.cc:492] T 3843ac68594f43079a1253a877c48065 P 21e73dc1a0eb486db9717fa320347e1c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 21:25:56.173799 1738 tablet_bootstrap.cc:492] T 3843ac68594f43079a1253a877c48065 P 21e73dc1a0eb486db9717fa320347e1c: Bootstrap complete.
I20240502 21:25:56.174212 1738 ts_tablet_manager.cc:1367] T 3843ac68594f43079a1253a877c48065 P 21e73dc1a0eb486db9717fa320347e1c: Time spent bootstrapping tablet: real 0.041s user 0.040s sys 0.000s
I20240502 21:25:56.178711 1738 raft_consensus.cc:348] T 3843ac68594f43079a1253a877c48065 P 21e73dc1a0eb486db9717fa320347e1c [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:56.179627 1738 raft_consensus.cc:729] T 3843ac68594f43079a1253a877c48065 P 21e73dc1a0eb486db9717fa320347e1c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21e73dc1a0eb486db9717fa320347e1c, State: Initialized, Role: FOLLOWER
I20240502 21:25:56.180070 1738 consensus_queue.cc:260] T 3843ac68594f43079a1253a877c48065 P 21e73dc1a0eb486db9717fa320347e1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "21e73dc1a0eb486db9717fa320347e1c" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 36679 } }
I20240502 21:25:56.181540 1738 ts_tablet_manager.cc:1398] T 3843ac68594f43079a1253a877c48065 P 21e73dc1a0eb486db9717fa320347e1c: Time spent starting tablet: real 0.007s user 0.005s sys 0.004s
I20240502 21:25:56.182526 32283 tablet_copy_service.cc:342] P f6f8b5e2067c4c23b0c89a8c471626f5: Request end of tablet copy session 21e73dc1a0eb486db9717fa320347e1c-3843ac68594f43079a1253a877c48065 received from {username='slave'} at 127.29.82.69:44845
I20240502 21:25:56.182663 32283 tablet_copy_service.cc:433] P f6f8b5e2067c4c23b0c89a8c471626f5: ending tablet copy session 21e73dc1a0eb486db9717fa320347e1c-3843ac68594f43079a1253a877c48065 on tablet 3843ac68594f43079a1253a877c48065 with peer 21e73dc1a0eb486db9717fa320347e1c
I20240502 21:25:56.513396 1177 raft_consensus.cc:1203] T 3843ac68594f43079a1253a877c48065 P 21e73dc1a0eb486db9717fa320347e1c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:25:56.515380 1746 raft_consensus.cc:2932] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 21e73dc1a0eb486db9717fa320347e1c (127.29.82.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "21e73dc1a0eb486db9717fa320347e1c" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 36679 } } }
I20240502 21:25:56.516253 1177 raft_consensus.cc:2932] T 3843ac68594f43079a1253a877c48065 P 21e73dc1a0eb486db9717fa320347e1c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 21e73dc1a0eb486db9717fa320347e1c (127.29.82.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "21e73dc1a0eb486db9717fa320347e1c" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 36679 } } }
I20240502 21:25:56.521072 31978 catalog_manager.cc:5484] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5 reported cstate change: config changed from index -1 to 4, VOTER 21e73dc1a0eb486db9717fa320347e1c (127.29.82.69) added. New cstate: current_term: 1 leader_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "21e73dc1a0eb486db9717fa320347e1c" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 36679 } } }
I20240502 21:25:56.554275 32263 consensus_queue.cc:237] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "98ac0bd5300e42c38bee7aae9634c7c9" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 39493 } }
W20240502 21:25:56.562366 32194 consensus_peers.cc:485] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5 -> Peer 98ac0bd5300e42c38bee7aae9634c7c9 (127.29.82.71:39493): Couldn't send request to peer 98ac0bd5300e42c38bee7aae9634c7c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd16f492f22c478a946d1da05f6a91b4. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:56.999153 1770 ts_tablet_manager.cc:902] T cd16f492f22c478a946d1da05f6a91b4 P 98ac0bd5300e42c38bee7aae9634c7c9: Initiating tablet copy from peer f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66:39503)
I20240502 21:25:57.000633 1770 tablet_copy_client.cc:308] T cd16f492f22c478a946d1da05f6a91b4 P 98ac0bd5300e42c38bee7aae9634c7c9: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.66:39503
I20240502 21:25:57.006963 32283 tablet_copy_service.cc:140] P f6f8b5e2067c4c23b0c89a8c471626f5: Received BeginTabletCopySession request for tablet cd16f492f22c478a946d1da05f6a91b4 from peer 98ac0bd5300e42c38bee7aae9634c7c9 ({username='slave'} at 127.29.82.71:33475)
I20240502 21:25:57.007117 32283 tablet_copy_service.cc:161] P f6f8b5e2067c4c23b0c89a8c471626f5: Beginning new tablet copy session on tablet cd16f492f22c478a946d1da05f6a91b4 from peer 98ac0bd5300e42c38bee7aae9634c7c9 at {username='slave'} at 127.29.82.71:33475: session id = 98ac0bd5300e42c38bee7aae9634c7c9-cd16f492f22c478a946d1da05f6a91b4
I20240502 21:25:57.009124 32283 tablet_copy_source_session.cc:205] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:25:57.010430 1770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd16f492f22c478a946d1da05f6a91b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:57.017196 1770 tablet_copy_client.cc:783] T cd16f492f22c478a946d1da05f6a91b4 P 98ac0bd5300e42c38bee7aae9634c7c9: tablet copy: Starting download of 0 data blocks...
I20240502 21:25:57.017467 1770 tablet_copy_client.cc:647] T cd16f492f22c478a946d1da05f6a91b4 P 98ac0bd5300e42c38bee7aae9634c7c9: tablet copy: Starting download of 1 WAL segments...
I20240502 21:25:57.019308 1770 tablet_copy_client.cc:523] T cd16f492f22c478a946d1da05f6a91b4 P 98ac0bd5300e42c38bee7aae9634c7c9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:25:57.022023 1770 tablet_bootstrap.cc:492] T cd16f492f22c478a946d1da05f6a91b4 P 98ac0bd5300e42c38bee7aae9634c7c9: Bootstrap starting.
I20240502 21:25:57.053853 1770 tablet_bootstrap.cc:492] T cd16f492f22c478a946d1da05f6a91b4 P 98ac0bd5300e42c38bee7aae9634c7c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 21:25:57.054296 1770 tablet_bootstrap.cc:492] T cd16f492f22c478a946d1da05f6a91b4 P 98ac0bd5300e42c38bee7aae9634c7c9: Bootstrap complete.
I20240502 21:25:57.054554 1770 ts_tablet_manager.cc:1367] T cd16f492f22c478a946d1da05f6a91b4 P 98ac0bd5300e42c38bee7aae9634c7c9: Time spent bootstrapping tablet: real 0.033s user 0.024s sys 0.008s
I20240502 21:25:57.055533 1770 raft_consensus.cc:348] T cd16f492f22c478a946d1da05f6a91b4 P 98ac0bd5300e42c38bee7aae9634c7c9 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:57.056090 1770 raft_consensus.cc:729] T cd16f492f22c478a946d1da05f6a91b4 P 98ac0bd5300e42c38bee7aae9634c7c9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98ac0bd5300e42c38bee7aae9634c7c9, State: Initialized, Role: FOLLOWER
I20240502 21:25:57.056283 1770 consensus_queue.cc:260] T cd16f492f22c478a946d1da05f6a91b4 P 98ac0bd5300e42c38bee7aae9634c7c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "98ac0bd5300e42c38bee7aae9634c7c9" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 39493 } }
I20240502 21:25:57.056962 1770 ts_tablet_manager.cc:1398] T cd16f492f22c478a946d1da05f6a91b4 P 98ac0bd5300e42c38bee7aae9634c7c9: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240502 21:25:57.057847 32283 tablet_copy_service.cc:342] P f6f8b5e2067c4c23b0c89a8c471626f5: Request end of tablet copy session 98ac0bd5300e42c38bee7aae9634c7c9-cd16f492f22c478a946d1da05f6a91b4 received from {username='slave'} at 127.29.82.71:33475
I20240502 21:25:57.058017 32283 tablet_copy_service.cc:433] P f6f8b5e2067c4c23b0c89a8c471626f5: ending tablet copy session 98ac0bd5300e42c38bee7aae9634c7c9-cd16f492f22c478a946d1da05f6a91b4 on tablet cd16f492f22c478a946d1da05f6a91b4 with peer 98ac0bd5300e42c38bee7aae9634c7c9
I20240502 21:25:57.574615 1440 raft_consensus.cc:1203] T cd16f492f22c478a946d1da05f6a91b4 P 98ac0bd5300e42c38bee7aae9634c7c9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:25:57.576117 1779 raft_consensus.cc:2932] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 98ac0bd5300e42c38bee7aae9634c7c9 (127.29.82.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "98ac0bd5300e42c38bee7aae9634c7c9" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 39493 } } }
I20240502 21:25:57.576797 1440 raft_consensus.cc:2932] T cd16f492f22c478a946d1da05f6a91b4 P 98ac0bd5300e42c38bee7aae9634c7c9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 98ac0bd5300e42c38bee7aae9634c7c9 (127.29.82.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "98ac0bd5300e42c38bee7aae9634c7c9" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 39493 } } }
I20240502 21:25:57.579934 1779 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:33490 (request call id 0) took 1026 ms. Trace:
I20240502 21:25:57.580207 1779 rpcz_store.cc:276] 0502 21:25:56.553757 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:25:56.553845 (+ 88us) service_pool.cc:225] Handling call
0502 21:25:56.554943 (+ 1098us) log.cc:842] Serialized 257 byte log entry
0502 21:25:57.579874 (+1024931us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":85,"raft.queue_time_us":66,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":280,"spinlock_wait_cycles":40832,"thread_start_us":214,"threads_started":1,"wal-append.queue_time_us":341,"wal-append.run_cpu_time_us":698,"wal-append.run_wall_time_us":1000734}
I20240502 21:25:57.582260 31971 catalog_manager.cc:5484] T cd16f492f22c478a946d1da05f6a91b4 P 98ac0bd5300e42c38bee7aae9634c7c9 reported cstate change: config changed from index -1 to 4, VOTER 98ac0bd5300e42c38bee7aae9634c7c9 (127.29.82.71) added. New cstate: current_term: 1 leader_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "98ac0bd5300e42c38bee7aae9634c7c9" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 39493 } } }
I20240502 21:25:57.618784 32127 consensus_queue.cc:237] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "f58850eb82214ad39880e45777bf51b4" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 33951 } }
W20240502 21:25:57.626703 32059 consensus_peers.cc:485] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9 -> Peer f58850eb82214ad39880e45777bf51b4 (127.29.82.68:33951): Couldn't send request to peer f58850eb82214ad39880e45777bf51b4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9812cd5d31154807a4f605eb615553f1. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:57.996480 1807 ts_tablet_manager.cc:902] T 9812cd5d31154807a4f605eb615553f1 P f58850eb82214ad39880e45777bf51b4: Initiating tablet copy from peer cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65:37947)
I20240502 21:25:57.997713 1807 tablet_copy_client.cc:308] T 9812cd5d31154807a4f605eb615553f1 P f58850eb82214ad39880e45777bf51b4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.65:37947
I20240502 21:25:58.003629 32147 tablet_copy_service.cc:140] P cb8caceef23e475f8304b1ec1175e4b9: Received BeginTabletCopySession request for tablet 9812cd5d31154807a4f605eb615553f1 from peer f58850eb82214ad39880e45777bf51b4 ({username='slave'} at 127.29.82.68:35719)
I20240502 21:25:58.003774 32147 tablet_copy_service.cc:161] P cb8caceef23e475f8304b1ec1175e4b9: Beginning new tablet copy session on tablet 9812cd5d31154807a4f605eb615553f1 from peer f58850eb82214ad39880e45777bf51b4 at {username='slave'} at 127.29.82.68:35719: session id = f58850eb82214ad39880e45777bf51b4-9812cd5d31154807a4f605eb615553f1
I20240502 21:25:58.005663 32147 tablet_copy_source_session.cc:205] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:25:58.006790 1807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9812cd5d31154807a4f605eb615553f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:58.011116 1807 tablet_copy_client.cc:783] T 9812cd5d31154807a4f605eb615553f1 P f58850eb82214ad39880e45777bf51b4: tablet copy: Starting download of 0 data blocks...
I20240502 21:25:58.011401 1807 tablet_copy_client.cc:647] T 9812cd5d31154807a4f605eb615553f1 P f58850eb82214ad39880e45777bf51b4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:25:58.013165 1807 tablet_copy_client.cc:523] T 9812cd5d31154807a4f605eb615553f1 P f58850eb82214ad39880e45777bf51b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:25:58.016036 1807 tablet_bootstrap.cc:492] T 9812cd5d31154807a4f605eb615553f1 P f58850eb82214ad39880e45777bf51b4: Bootstrap starting.
I20240502 21:25:58.020511 1807 tablet_bootstrap.cc:492] T 9812cd5d31154807a4f605eb615553f1 P f58850eb82214ad39880e45777bf51b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 21:25:58.020881 1807 tablet_bootstrap.cc:492] T 9812cd5d31154807a4f605eb615553f1 P f58850eb82214ad39880e45777bf51b4: Bootstrap complete.
I20240502 21:25:58.021090 1807 ts_tablet_manager.cc:1367] T 9812cd5d31154807a4f605eb615553f1 P f58850eb82214ad39880e45777bf51b4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:25:58.021970 1807 raft_consensus.cc:348] T 9812cd5d31154807a4f605eb615553f1 P f58850eb82214ad39880e45777bf51b4 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:58.022370 1807 raft_consensus.cc:729] T 9812cd5d31154807a4f605eb615553f1 P f58850eb82214ad39880e45777bf51b4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58850eb82214ad39880e45777bf51b4, State: Initialized, Role: FOLLOWER
I20240502 21:25:58.022584 1807 consensus_queue.cc:260] T 9812cd5d31154807a4f605eb615553f1 P f58850eb82214ad39880e45777bf51b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "f58850eb82214ad39880e45777bf51b4" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 33951 } }
I20240502 21:25:58.023452 1807 ts_tablet_manager.cc:1398] T 9812cd5d31154807a4f605eb615553f1 P f58850eb82214ad39880e45777bf51b4: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20240502 21:25:58.024277 32147 tablet_copy_service.cc:342] P cb8caceef23e475f8304b1ec1175e4b9: Request end of tablet copy session f58850eb82214ad39880e45777bf51b4-9812cd5d31154807a4f605eb615553f1 received from {username='slave'} at 127.29.82.68:35719
I20240502 21:25:58.024387 32147 tablet_copy_service.cc:433] P cb8caceef23e475f8304b1ec1175e4b9: ending tablet copy session f58850eb82214ad39880e45777bf51b4-9812cd5d31154807a4f605eb615553f1 on tablet 9812cd5d31154807a4f605eb615553f1 with peer f58850eb82214ad39880e45777bf51b4
I20240502 21:25:58.375116 1046 raft_consensus.cc:1203] T 9812cd5d31154807a4f605eb615553f1 P f58850eb82214ad39880e45777bf51b4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:25:58.376345 1815 raft_consensus.cc:2932] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER f58850eb82214ad39880e45777bf51b4 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "f58850eb82214ad39880e45777bf51b4" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 33951 } } }
I20240502 21:25:58.376962 1046 raft_consensus.cc:2932] T 9812cd5d31154807a4f605eb615553f1 P f58850eb82214ad39880e45777bf51b4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER f58850eb82214ad39880e45777bf51b4 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "f58850eb82214ad39880e45777bf51b4" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 33951 } } }
I20240502 21:25:58.382194 31978 catalog_manager.cc:5484] T 9812cd5d31154807a4f605eb615553f1 P f58850eb82214ad39880e45777bf51b4 reported cstate change: config changed from index -1 to 2, VOTER f58850eb82214ad39880e45777bf51b4 (127.29.82.68) added. New cstate: current_term: 1 leader_uuid: "cb8caceef23e475f8304b1ec1175e4b9" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "f58850eb82214ad39880e45777bf51b4" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 33951 } } }
I20240502 21:25:58.420773 32263 consensus_queue.cc:237] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "b399bf9ec0c544749c7eead22ddfedf6" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 45329 } }
W20240502 21:25:58.429049 32196 consensus_peers.cc:485] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5 -> Peer b399bf9ec0c544749c7eead22ddfedf6 (127.29.82.67:45329): Couldn't send request to peer b399bf9ec0c544749c7eead22ddfedf6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 43d4ba4bfe70416bb9c06a3bfe299835. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:58.897471 1840 ts_tablet_manager.cc:902] T 43d4ba4bfe70416bb9c06a3bfe299835 P b399bf9ec0c544749c7eead22ddfedf6: Initiating tablet copy from peer f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66:39503)
I20240502 21:25:58.898758 1840 tablet_copy_client.cc:308] T 43d4ba4bfe70416bb9c06a3bfe299835 P b399bf9ec0c544749c7eead22ddfedf6: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.66:39503
I20240502 21:25:58.906440 32283 tablet_copy_service.cc:140] P f6f8b5e2067c4c23b0c89a8c471626f5: Received BeginTabletCopySession request for tablet 43d4ba4bfe70416bb9c06a3bfe299835 from peer b399bf9ec0c544749c7eead22ddfedf6 ({username='slave'} at 127.29.82.67:52961)
I20240502 21:25:58.906610 32283 tablet_copy_service.cc:161] P f6f8b5e2067c4c23b0c89a8c471626f5: Beginning new tablet copy session on tablet 43d4ba4bfe70416bb9c06a3bfe299835 from peer b399bf9ec0c544749c7eead22ddfedf6 at {username='slave'} at 127.29.82.67:52961: session id = b399bf9ec0c544749c7eead22ddfedf6-43d4ba4bfe70416bb9c06a3bfe299835
I20240502 21:25:58.908627 32283 tablet_copy_source_session.cc:205] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:25:58.910037 1840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43d4ba4bfe70416bb9c06a3bfe299835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:58.916617 1840 tablet_copy_client.cc:783] T 43d4ba4bfe70416bb9c06a3bfe299835 P b399bf9ec0c544749c7eead22ddfedf6: tablet copy: Starting download of 0 data blocks...
I20240502 21:25:58.916926 1840 tablet_copy_client.cc:647] T 43d4ba4bfe70416bb9c06a3bfe299835 P b399bf9ec0c544749c7eead22ddfedf6: tablet copy: Starting download of 1 WAL segments...
I20240502 21:25:58.918797 1840 tablet_copy_client.cc:523] T 43d4ba4bfe70416bb9c06a3bfe299835 P b399bf9ec0c544749c7eead22ddfedf6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:25:58.922382 1840 tablet_bootstrap.cc:492] T 43d4ba4bfe70416bb9c06a3bfe299835 P b399bf9ec0c544749c7eead22ddfedf6: Bootstrap starting.
I20240502 21:25:58.954788 1840 tablet_bootstrap.cc:492] T 43d4ba4bfe70416bb9c06a3bfe299835 P b399bf9ec0c544749c7eead22ddfedf6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 21:25:58.955222 1840 tablet_bootstrap.cc:492] T 43d4ba4bfe70416bb9c06a3bfe299835 P b399bf9ec0c544749c7eead22ddfedf6: Bootstrap complete.
I20240502 21:25:58.955483 1840 ts_tablet_manager.cc:1367] T 43d4ba4bfe70416bb9c06a3bfe299835 P b399bf9ec0c544749c7eead22ddfedf6: Time spent bootstrapping tablet: real 0.033s user 0.028s sys 0.004s
I20240502 21:25:58.956422 1840 raft_consensus.cc:348] T 43d4ba4bfe70416bb9c06a3bfe299835 P b399bf9ec0c544749c7eead22ddfedf6 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:25:58.956748 1840 raft_consensus.cc:729] T 43d4ba4bfe70416bb9c06a3bfe299835 P b399bf9ec0c544749c7eead22ddfedf6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b399bf9ec0c544749c7eead22ddfedf6, State: Initialized, Role: FOLLOWER
I20240502 21:25:58.956889 1840 consensus_queue.cc:260] T 43d4ba4bfe70416bb9c06a3bfe299835 P b399bf9ec0c544749c7eead22ddfedf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "b399bf9ec0c544749c7eead22ddfedf6" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 45329 } }
I20240502 21:25:58.957532 1840 ts_tablet_manager.cc:1398] T 43d4ba4bfe70416bb9c06a3bfe299835 P b399bf9ec0c544749c7eead22ddfedf6: Time spent starting tablet: real 0.002s user 0.006s sys 0.000s
I20240502 21:25:58.958421 32283 tablet_copy_service.cc:342] P f6f8b5e2067c4c23b0c89a8c471626f5: Request end of tablet copy session b399bf9ec0c544749c7eead22ddfedf6-43d4ba4bfe70416bb9c06a3bfe299835 received from {username='slave'} at 127.29.82.67:52961
I20240502 21:25:58.958564 32283 tablet_copy_service.cc:433] P f6f8b5e2067c4c23b0c89a8c471626f5: ending tablet copy session b399bf9ec0c544749c7eead22ddfedf6-43d4ba4bfe70416bb9c06a3bfe299835 on tablet 43d4ba4bfe70416bb9c06a3bfe299835 with peer b399bf9ec0c544749c7eead22ddfedf6
I20240502 21:25:59.314713 911 raft_consensus.cc:1203] T 43d4ba4bfe70416bb9c06a3bfe299835 P b399bf9ec0c544749c7eead22ddfedf6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:25:59.316186 1848 raft_consensus.cc:2932] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b399bf9ec0c544749c7eead22ddfedf6 (127.29.82.67) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "b399bf9ec0c544749c7eead22ddfedf6" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 45329 } } }
I20240502 21:25:59.316928 911 raft_consensus.cc:2932] T 43d4ba4bfe70416bb9c06a3bfe299835 P b399bf9ec0c544749c7eead22ddfedf6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b399bf9ec0c544749c7eead22ddfedf6 (127.29.82.67) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "b399bf9ec0c544749c7eead22ddfedf6" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 45329 } } }
I20240502 21:25:59.321717 31978 catalog_manager.cc:5484] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5 reported cstate change: config changed from index -1 to 4, VOTER b399bf9ec0c544749c7eead22ddfedf6 (127.29.82.67) added. New cstate: current_term: 1 leader_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "b399bf9ec0c544749c7eead22ddfedf6" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 45329 } } }
I20240502 21:25:59.353506 32127 consensus_queue.cc:237] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "e705f0dda7904364a7453ee6294b3601" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 45931 } }
W20240502 21:25:59.356026 32062 consensus_peers.cc:485] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9 -> Peer e705f0dda7904364a7453ee6294b3601 (127.29.82.70:45931): Couldn't send request to peer e705f0dda7904364a7453ee6294b3601. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9873952de1224793b8b7f17ef92295b7. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:59.875977 1871 ts_tablet_manager.cc:902] T 9873952de1224793b8b7f17ef92295b7 P e705f0dda7904364a7453ee6294b3601: Initiating tablet copy from peer cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65:37947)
I20240502 21:25:59.877126 1871 tablet_copy_client.cc:308] T 9873952de1224793b8b7f17ef92295b7 P e705f0dda7904364a7453ee6294b3601: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.65:37947
I20240502 21:25:59.877852 32147 tablet_copy_service.cc:140] P cb8caceef23e475f8304b1ec1175e4b9: Received BeginTabletCopySession request for tablet 9873952de1224793b8b7f17ef92295b7 from peer e705f0dda7904364a7453ee6294b3601 ({username='slave'} at 127.29.82.70:53679)
I20240502 21:25:59.878010 32147 tablet_copy_service.cc:161] P cb8caceef23e475f8304b1ec1175e4b9: Beginning new tablet copy session on tablet 9873952de1224793b8b7f17ef92295b7 from peer e705f0dda7904364a7453ee6294b3601 at {username='slave'} at 127.29.82.70:53679: session id = e705f0dda7904364a7453ee6294b3601-9873952de1224793b8b7f17ef92295b7
I20240502 21:25:59.880012 32147 tablet_copy_source_session.cc:205] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:25:59.881131 1871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9873952de1224793b8b7f17ef92295b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:59.885668 1871 tablet_copy_client.cc:783] T 9873952de1224793b8b7f17ef92295b7 P e705f0dda7904364a7453ee6294b3601: tablet copy: Starting download of 0 data blocks...
I20240502 21:25:59.885906 1871 tablet_copy_client.cc:647] T 9873952de1224793b8b7f17ef92295b7 P e705f0dda7904364a7453ee6294b3601: tablet copy: Starting download of 1 WAL segments...
I20240502 21:25:59.887508 1871 tablet_copy_client.cc:523] T 9873952de1224793b8b7f17ef92295b7 P e705f0dda7904364a7453ee6294b3601: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:25:59.889775 1871 tablet_bootstrap.cc:492] T 9873952de1224793b8b7f17ef92295b7 P e705f0dda7904364a7453ee6294b3601: Bootstrap starting.
I20240502 21:25:59.919317 1871 tablet_bootstrap.cc:492] T 9873952de1224793b8b7f17ef92295b7 P e705f0dda7904364a7453ee6294b3601: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 21:25:59.919762 1871 tablet_bootstrap.cc:492] T 9873952de1224793b8b7f17ef92295b7 P e705f0dda7904364a7453ee6294b3601: Bootstrap complete.
I20240502 21:25:59.920045 1871 ts_tablet_manager.cc:1367] T 9873952de1224793b8b7f17ef92295b7 P e705f0dda7904364a7453ee6294b3601: Time spent bootstrapping tablet: real 0.030s user 0.024s sys 0.008s
I20240502 21:25:59.920897 1871 raft_consensus.cc:348] T 9873952de1224793b8b7f17ef92295b7 P e705f0dda7904364a7453ee6294b3601 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:25:59.921327 1871 raft_consensus.cc:729] T 9873952de1224793b8b7f17ef92295b7 P e705f0dda7904364a7453ee6294b3601 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e705f0dda7904364a7453ee6294b3601, State: Initialized, Role: FOLLOWER
I20240502 21:25:59.921525 1871 consensus_queue.cc:260] T 9873952de1224793b8b7f17ef92295b7 P e705f0dda7904364a7453ee6294b3601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "e705f0dda7904364a7453ee6294b3601" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 45931 } }
I20240502 21:25:59.922351 1871 ts_tablet_manager.cc:1398] T 9873952de1224793b8b7f17ef92295b7 P e705f0dda7904364a7453ee6294b3601: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:59.923285 32147 tablet_copy_service.cc:342] P cb8caceef23e475f8304b1ec1175e4b9: Request end of tablet copy session e705f0dda7904364a7453ee6294b3601-9873952de1224793b8b7f17ef92295b7 received from {username='slave'} at 127.29.82.70:53679
I20240502 21:25:59.923447 32147 tablet_copy_service.cc:433] P cb8caceef23e475f8304b1ec1175e4b9: ending tablet copy session e705f0dda7904364a7453ee6294b3601-9873952de1224793b8b7f17ef92295b7 on tablet 9873952de1224793b8b7f17ef92295b7 with peer e705f0dda7904364a7453ee6294b3601
I20240502 21:26:00.432590 1308 raft_consensus.cc:1203] T 9873952de1224793b8b7f17ef92295b7 P e705f0dda7904364a7453ee6294b3601 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:26:00.434129 1879 raft_consensus.cc:2932] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e705f0dda7904364a7453ee6294b3601 (127.29.82.70) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "e705f0dda7904364a7453ee6294b3601" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 45931 } } }
I20240502 21:26:00.434832 1308 raft_consensus.cc:2932] T 9873952de1224793b8b7f17ef92295b7 P e705f0dda7904364a7453ee6294b3601 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e705f0dda7904364a7453ee6294b3601 (127.29.82.70) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "e705f0dda7904364a7453ee6294b3601" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 45931 } } }
I20240502 21:26:00.438992 1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:50096 (request call id 0) took 1086 ms. Trace:
I20240502 21:26:00.439181 1879 rpcz_store.cc:276] 0502 21:25:59.352777 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:25:59.352870 (+ 93us) service_pool.cc:225] Handling call
0502 21:25:59.354267 (+ 1397us) log.cc:842] Serialized 257 byte log entry
0502 21:26:00.438939 (+1084672us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":71,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":211,"spinlock_wait_cycles":2304,"thread_start_us":228,"threads_started":1,"wal-append.queue_time_us":376,"wal-append.run_cpu_time_us":558,"wal-append.run_wall_time_us":1000611}
I20240502 21:26:00.440603 31978 catalog_manager.cc:5484] T 9873952de1224793b8b7f17ef92295b7 P e705f0dda7904364a7453ee6294b3601 reported cstate change: config changed from index -1 to 4, VOTER e705f0dda7904364a7453ee6294b3601 (127.29.82.70) added. New cstate: current_term: 1 leader_uuid: "cb8caceef23e475f8304b1ec1175e4b9" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "e705f0dda7904364a7453ee6294b3601" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 45931 } } }
I20240502 21:26:00.475149 32263 consensus_queue.cc:237] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "21e73dc1a0eb486db9717fa320347e1c" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 36679 } }
W20240502 21:26:00.477337 32194 consensus_peers.cc:485] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5 -> Peer 21e73dc1a0eb486db9717fa320347e1c (127.29.82.69:36679): Couldn't send request to peer 21e73dc1a0eb486db9717fa320347e1c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a80b4f90d4b04e1886e494cf23a6284b. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:00.974529 1905 ts_tablet_manager.cc:902] T a80b4f90d4b04e1886e494cf23a6284b P 21e73dc1a0eb486db9717fa320347e1c: Initiating tablet copy from peer f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66:39503)
I20240502 21:26:00.975956 1905 tablet_copy_client.cc:308] T a80b4f90d4b04e1886e494cf23a6284b P 21e73dc1a0eb486db9717fa320347e1c: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.66:39503
I20240502 21:26:00.976727 32283 tablet_copy_service.cc:140] P f6f8b5e2067c4c23b0c89a8c471626f5: Received BeginTabletCopySession request for tablet a80b4f90d4b04e1886e494cf23a6284b from peer 21e73dc1a0eb486db9717fa320347e1c ({username='slave'} at 127.29.82.69:44845)
I20240502 21:26:00.976893 32283 tablet_copy_service.cc:161] P f6f8b5e2067c4c23b0c89a8c471626f5: Beginning new tablet copy session on tablet a80b4f90d4b04e1886e494cf23a6284b from peer 21e73dc1a0eb486db9717fa320347e1c at {username='slave'} at 127.29.82.69:44845: session id = 21e73dc1a0eb486db9717fa320347e1c-a80b4f90d4b04e1886e494cf23a6284b
I20240502 21:26:00.979387 32283 tablet_copy_source_session.cc:205] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:00.980620 1905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a80b4f90d4b04e1886e494cf23a6284b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:00.985112 1905 tablet_copy_client.cc:783] T a80b4f90d4b04e1886e494cf23a6284b P 21e73dc1a0eb486db9717fa320347e1c: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:00.985354 1905 tablet_copy_client.cc:647] T a80b4f90d4b04e1886e494cf23a6284b P 21e73dc1a0eb486db9717fa320347e1c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:00.986986 1905 tablet_copy_client.cc:523] T a80b4f90d4b04e1886e494cf23a6284b P 21e73dc1a0eb486db9717fa320347e1c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:00.989485 1905 tablet_bootstrap.cc:492] T a80b4f90d4b04e1886e494cf23a6284b P 21e73dc1a0eb486db9717fa320347e1c: Bootstrap starting.
I20240502 21:26:01.022215 1905 tablet_bootstrap.cc:492] T a80b4f90d4b04e1886e494cf23a6284b P 21e73dc1a0eb486db9717fa320347e1c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 21:26:01.022783 1905 tablet_bootstrap.cc:492] T a80b4f90d4b04e1886e494cf23a6284b P 21e73dc1a0eb486db9717fa320347e1c: Bootstrap complete.
I20240502 21:26:01.023108 1905 ts_tablet_manager.cc:1367] T a80b4f90d4b04e1886e494cf23a6284b P 21e73dc1a0eb486db9717fa320347e1c: Time spent bootstrapping tablet: real 0.034s user 0.029s sys 0.006s
I20240502 21:26:01.024209 1905 raft_consensus.cc:348] T a80b4f90d4b04e1886e494cf23a6284b P 21e73dc1a0eb486db9717fa320347e1c [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:26:01.024705 1905 raft_consensus.cc:729] T a80b4f90d4b04e1886e494cf23a6284b P 21e73dc1a0eb486db9717fa320347e1c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21e73dc1a0eb486db9717fa320347e1c, State: Initialized, Role: FOLLOWER
I20240502 21:26:01.024919 1905 consensus_queue.cc:260] T a80b4f90d4b04e1886e494cf23a6284b P 21e73dc1a0eb486db9717fa320347e1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "21e73dc1a0eb486db9717fa320347e1c" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 36679 } }
I20240502 21:26:01.025790 1905 ts_tablet_manager.cc:1398] T a80b4f90d4b04e1886e494cf23a6284b P 21e73dc1a0eb486db9717fa320347e1c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:01.026744 32283 tablet_copy_service.cc:342] P f6f8b5e2067c4c23b0c89a8c471626f5: Request end of tablet copy session 21e73dc1a0eb486db9717fa320347e1c-a80b4f90d4b04e1886e494cf23a6284b received from {username='slave'} at 127.29.82.69:44845
I20240502 21:26:01.026912 32283 tablet_copy_service.cc:433] P f6f8b5e2067c4c23b0c89a8c471626f5: ending tablet copy session 21e73dc1a0eb486db9717fa320347e1c-a80b4f90d4b04e1886e494cf23a6284b on tablet a80b4f90d4b04e1886e494cf23a6284b with peer 21e73dc1a0eb486db9717fa320347e1c
I20240502 21:26:01.380090 1177 raft_consensus.cc:1203] T a80b4f90d4b04e1886e494cf23a6284b P 21e73dc1a0eb486db9717fa320347e1c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:26:01.381718 1912 raft_consensus.cc:2932] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 21e73dc1a0eb486db9717fa320347e1c (127.29.82.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "21e73dc1a0eb486db9717fa320347e1c" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 36679 } } }
I20240502 21:26:01.382480 1177 raft_consensus.cc:2932] T a80b4f90d4b04e1886e494cf23a6284b P 21e73dc1a0eb486db9717fa320347e1c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 21e73dc1a0eb486db9717fa320347e1c (127.29.82.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "21e73dc1a0eb486db9717fa320347e1c" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 36679 } } }
I20240502 21:26:01.387431 31971 catalog_manager.cc:5484] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5 reported cstate change: config changed from index -1 to 4, VOTER 21e73dc1a0eb486db9717fa320347e1c (127.29.82.69) added. New cstate: current_term: 1 leader_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "21e73dc1a0eb486db9717fa320347e1c" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 36679 } } }
I20240502 21:26:01.493921 1420 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:01.502672 1157 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:01.503852 890 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:01.528035 32107 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:26:01.561338 1024 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:01.562880 1288 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:01.585808 32242 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:26:01.640815 32127 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9873952de1224793b8b7f17ef92295b7"
dest_uuid: "cb8caceef23e475f8304b1ec1175e4b9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50138
I20240502 21:26:01.641120 32127 raft_consensus.cc:595] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:26:01.733487 1157 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:01.742821 1420 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:01.757642 32107 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:26:01.757902 890 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:01.760735 1288 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:01.773391 1024 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:01.776911 32242 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:26:01.940801 2025 raft_consensus.cc:979] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9: : Instructing follower e705f0dda7904364a7453ee6294b3601 to start an election
I20240502 21:26:01.941012 2025 raft_consensus.cc:1067] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Signalling peer e705f0dda7904364a7453ee6294b3601 to start an election
I20240502 21:26:01.942145 1308 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9873952de1224793b8b7f17ef92295b7"
dest_uuid: "e705f0dda7904364a7453ee6294b3601"
from {username='slave'} at 127.29.82.65:38879
I20240502 21:26:01.942466 1308 raft_consensus.cc:482] T 9873952de1224793b8b7f17ef92295b7 P e705f0dda7904364a7453ee6294b3601 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:26:01.942663 1308 raft_consensus.cc:3037] T 9873952de1224793b8b7f17ef92295b7 P e705f0dda7904364a7453ee6294b3601 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:01.945619 1308 raft_consensus.cc:504] T 9873952de1224793b8b7f17ef92295b7 P e705f0dda7904364a7453ee6294b3601 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "e705f0dda7904364a7453ee6294b3601" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 45931 } }
I20240502 21:26:01.947166 1308 leader_election.cc:288] T 9873952de1224793b8b7f17ef92295b7 P e705f0dda7904364a7453ee6294b3601 [CANDIDATE]: Term 2 election: Requested vote from peers cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65:37947)
I20240502 21:26:01.947778 32127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9873952de1224793b8b7f17ef92295b7" candidate_uuid: "e705f0dda7904364a7453ee6294b3601" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "cb8caceef23e475f8304b1ec1175e4b9"
I20240502 21:26:01.948094 32127 raft_consensus.cc:3032] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:26:01.948242 32127 raft_consensus.cc:729] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cb8caceef23e475f8304b1ec1175e4b9, State: Running, Role: LEADER
I20240502 21:26:01.948597 32127 consensus_queue.cc:260] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "e705f0dda7904364a7453ee6294b3601" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 45931 } }
I20240502 21:26:01.949172 32127 raft_consensus.cc:3037] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:01.954869 32127 raft_consensus.cc:2454] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e705f0dda7904364a7453ee6294b3601 in term 2.
I20240502 21:26:01.955881 1244 leader_election.cc:302] T 9873952de1224793b8b7f17ef92295b7 P e705f0dda7904364a7453ee6294b3601 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: cb8caceef23e475f8304b1ec1175e4b9, e705f0dda7904364a7453ee6294b3601; no voters:
I20240502 21:26:01.967063 1157 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:01.969920 2031 raft_consensus.cc:2781] T 9873952de1224793b8b7f17ef92295b7 P e705f0dda7904364a7453ee6294b3601 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:26:01.972155 1420 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:01.989751 890 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:02.003348 2031 raft_consensus.cc:686] T 9873952de1224793b8b7f17ef92295b7 P e705f0dda7904364a7453ee6294b3601 [term 2 LEADER]: Becoming Leader. State: Replica: e705f0dda7904364a7453ee6294b3601, State: Running, Role: LEADER
I20240502 21:26:02.004151 2031 consensus_queue.cc:237] T 9873952de1224793b8b7f17ef92295b7 P e705f0dda7904364a7453ee6294b3601 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "e705f0dda7904364a7453ee6294b3601" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 45931 } }
I20240502 21:26:02.016168 32107 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:26:02.044037 31980 catalog_manager.cc:5484] T 9873952de1224793b8b7f17ef92295b7 P e705f0dda7904364a7453ee6294b3601 reported cstate change: term changed from 1 to 2, leader changed from cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65) to e705f0dda7904364a7453ee6294b3601 (127.29.82.70). New cstate: current_term: 2 leader_uuid: "e705f0dda7904364a7453ee6294b3601" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "e705f0dda7904364a7453ee6294b3601" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 45931 } } }
I20240502 21:26:02.046859 32242 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:26:02.060616 1024 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:02.061576 1288 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:02.126082 32263 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "43d4ba4bfe70416bb9c06a3bfe299835"
dest_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33558
I20240502 21:26:02.126374 32263 raft_consensus.cc:595] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:26:02.232427 1157 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:02.233461 32107 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:26:02.237823 1420 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:02.240231 1288 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:02.246379 890 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:02.288039 32242 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:26:02.289206 1024 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:02.308765 1912 raft_consensus.cc:979] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5: : Instructing follower b399bf9ec0c544749c7eead22ddfedf6 to start an election
I20240502 21:26:02.308990 1912 raft_consensus.cc:1067] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Signalling peer b399bf9ec0c544749c7eead22ddfedf6 to start an election
I20240502 21:26:02.311899 911 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "43d4ba4bfe70416bb9c06a3bfe299835"
dest_uuid: "b399bf9ec0c544749c7eead22ddfedf6"
from {username='slave'} at 127.29.82.66:51395
I20240502 21:26:02.312255 911 raft_consensus.cc:482] T 43d4ba4bfe70416bb9c06a3bfe299835 P b399bf9ec0c544749c7eead22ddfedf6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:26:02.312505 911 raft_consensus.cc:3037] T 43d4ba4bfe70416bb9c06a3bfe299835 P b399bf9ec0c544749c7eead22ddfedf6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:02.315856 911 raft_consensus.cc:504] T 43d4ba4bfe70416bb9c06a3bfe299835 P b399bf9ec0c544749c7eead22ddfedf6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "b399bf9ec0c544749c7eead22ddfedf6" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 45329 } }
I20240502 21:26:02.317967 911 leader_election.cc:288] T 43d4ba4bfe70416bb9c06a3bfe299835 P b399bf9ec0c544749c7eead22ddfedf6 [CANDIDATE]: Term 2 election: Requested vote from peers f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66:39503)
I20240502 21:26:02.318151 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43d4ba4bfe70416bb9c06a3bfe299835" candidate_uuid: "b399bf9ec0c544749c7eead22ddfedf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5"
I20240502 21:26:02.318488 32263 raft_consensus.cc:3032] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:26:02.318583 32263 raft_consensus.cc:729] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6f8b5e2067c4c23b0c89a8c471626f5, State: Running, Role: LEADER
I20240502 21:26:02.318903 32263 consensus_queue.cc:260] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "b399bf9ec0c544749c7eead22ddfedf6" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 45329 } }
I20240502 21:26:02.319358 32263 raft_consensus.cc:3037] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:02.322788 32263 raft_consensus.cc:2454] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b399bf9ec0c544749c7eead22ddfedf6 in term 2.
I20240502 21:26:02.323518 843 leader_election.cc:302] T 43d4ba4bfe70416bb9c06a3bfe299835 P b399bf9ec0c544749c7eead22ddfedf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b399bf9ec0c544749c7eead22ddfedf6, f6f8b5e2067c4c23b0c89a8c471626f5; no voters:
I20240502 21:26:02.324306 2080 raft_consensus.cc:2781] T 43d4ba4bfe70416bb9c06a3bfe299835 P b399bf9ec0c544749c7eead22ddfedf6 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:26:02.335534 2080 raft_consensus.cc:686] T 43d4ba4bfe70416bb9c06a3bfe299835 P b399bf9ec0c544749c7eead22ddfedf6 [term 2 LEADER]: Becoming Leader. State: Replica: b399bf9ec0c544749c7eead22ddfedf6, State: Running, Role: LEADER
I20240502 21:26:02.336163 2080 consensus_queue.cc:237] T 43d4ba4bfe70416bb9c06a3bfe299835 P b399bf9ec0c544749c7eead22ddfedf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "b399bf9ec0c544749c7eead22ddfedf6" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 45329 } }
I20240502 21:26:02.340188 31974 catalog_manager.cc:5484] T 43d4ba4bfe70416bb9c06a3bfe299835 P b399bf9ec0c544749c7eead22ddfedf6 reported cstate change: term changed from 1 to 2, leader changed from f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66) to b399bf9ec0c544749c7eead22ddfedf6 (127.29.82.67). New cstate: current_term: 2 leader_uuid: "b399bf9ec0c544749c7eead22ddfedf6" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "b399bf9ec0c544749c7eead22ddfedf6" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 45329 } } }
I20240502 21:26:02.455471 32127 raft_consensus.cc:1261] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9 [term 2 FOLLOWER]: Refusing update from remote peer e705f0dda7904364a7453ee6294b3601: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:26:02.456431 2031 consensus_queue.cc:1035] T 9873952de1224793b8b7f17ef92295b7 P e705f0dda7904364a7453ee6294b3601 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:02.490622 1157 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:02.496205 890 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:02.499181 1420 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:02.504104 1288 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:02.510957 1024 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:02.522953 32107 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:26:02.523818 32242 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:26:02.610708 32127 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9812cd5d31154807a4f605eb615553f1"
dest_uuid: "cb8caceef23e475f8304b1ec1175e4b9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50186
I20240502 21:26:02.610917 32127 raft_consensus.cc:595] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:26:02.709832 1420 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:02.713428 1157 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:02.729097 32242 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:26:02.730849 32263 raft_consensus.cc:1261] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 2 FOLLOWER]: Refusing update from remote peer b399bf9ec0c544749c7eead22ddfedf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:26:02.732215 2080 consensus_queue.cc:1035] T 43d4ba4bfe70416bb9c06a3bfe299835 P b399bf9ec0c544749c7eead22ddfedf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:02.733389 890 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:02.734128 1024 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:02.734517 32107 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:26:02.764997 1288 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:02.832106 2173 raft_consensus.cc:979] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9: : Instructing follower f58850eb82214ad39880e45777bf51b4 to start an election
I20240502 21:26:02.832405 2173 raft_consensus.cc:1067] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Signalling peer f58850eb82214ad39880e45777bf51b4 to start an election
I20240502 21:26:02.833496 1046 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9812cd5d31154807a4f605eb615553f1"
dest_uuid: "f58850eb82214ad39880e45777bf51b4"
from {username='slave'} at 127.29.82.65:39871
I20240502 21:26:02.833815 1046 raft_consensus.cc:482] T 9812cd5d31154807a4f605eb615553f1 P f58850eb82214ad39880e45777bf51b4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:26:02.833990 1046 raft_consensus.cc:3037] T 9812cd5d31154807a4f605eb615553f1 P f58850eb82214ad39880e45777bf51b4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:02.837002 1046 raft_consensus.cc:504] T 9812cd5d31154807a4f605eb615553f1 P f58850eb82214ad39880e45777bf51b4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "f58850eb82214ad39880e45777bf51b4" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 33951 } }
I20240502 21:26:02.838644 1046 leader_election.cc:288] T 9812cd5d31154807a4f605eb615553f1 P f58850eb82214ad39880e45777bf51b4 [CANDIDATE]: Term 2 election: Requested vote from peers cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65:37947)
I20240502 21:26:02.839218 32127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9812cd5d31154807a4f605eb615553f1" candidate_uuid: "f58850eb82214ad39880e45777bf51b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "cb8caceef23e475f8304b1ec1175e4b9"
I20240502 21:26:02.839463 32127 raft_consensus.cc:3032] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:26:02.840633 32127 raft_consensus.cc:729] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cb8caceef23e475f8304b1ec1175e4b9, State: Running, Role: LEADER
I20240502 21:26:02.840981 32127 consensus_queue.cc:260] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "f58850eb82214ad39880e45777bf51b4" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 33951 } }
I20240502 21:26:02.841343 32127 raft_consensus.cc:3037] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:02.844602 32127 raft_consensus.cc:2454] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58850eb82214ad39880e45777bf51b4 in term 2.
I20240502 21:26:02.845362 980 leader_election.cc:302] T 9812cd5d31154807a4f605eb615553f1 P f58850eb82214ad39880e45777bf51b4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: cb8caceef23e475f8304b1ec1175e4b9, f58850eb82214ad39880e45777bf51b4; no voters:
I20240502 21:26:02.846202 2175 raft_consensus.cc:2781] T 9812cd5d31154807a4f605eb615553f1 P f58850eb82214ad39880e45777bf51b4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:26:02.846776 2175 raft_consensus.cc:686] T 9812cd5d31154807a4f605eb615553f1 P f58850eb82214ad39880e45777bf51b4 [term 2 LEADER]: Becoming Leader. State: Replica: f58850eb82214ad39880e45777bf51b4, State: Running, Role: LEADER
I20240502 21:26:02.847463 2175 consensus_queue.cc:237] T 9812cd5d31154807a4f605eb615553f1 P f58850eb82214ad39880e45777bf51b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "f58850eb82214ad39880e45777bf51b4" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 33951 } }
I20240502 21:26:02.852319 31978 catalog_manager.cc:5484] T 9812cd5d31154807a4f605eb615553f1 P f58850eb82214ad39880e45777bf51b4 reported cstate change: term changed from 1 to 2, leader changed from cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65) to f58850eb82214ad39880e45777bf51b4 (127.29.82.68). New cstate: current_term: 2 leader_uuid: "f58850eb82214ad39880e45777bf51b4" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "f58850eb82214ad39880e45777bf51b4" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 33951 } } }
I20240502 21:26:02.959898 1420 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:02.961874 890 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:02.962710 1157 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:02.967264 32107 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:26:02.977312 1288 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:02.993623 32242 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:26:03.003696 1024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:03.073032 32261 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "80471cfd4a944776baf05c4179f64a10"
dest_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33624
I20240502 21:26:03.073309 32261 raft_consensus.cc:595] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:26:03.166402 2249 raft_consensus.cc:979] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5: : Instructing follower f58850eb82214ad39880e45777bf51b4 to start an election
I20240502 21:26:03.166569 1912 raft_consensus.cc:1067] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Signalling peer f58850eb82214ad39880e45777bf51b4 to start an election
I20240502 21:26:03.167384 1046 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "80471cfd4a944776baf05c4179f64a10"
dest_uuid: "f58850eb82214ad39880e45777bf51b4"
from {username='slave'} at 127.29.82.66:34593
I20240502 21:26:03.167651 1046 raft_consensus.cc:482] T 80471cfd4a944776baf05c4179f64a10 P f58850eb82214ad39880e45777bf51b4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:26:03.167788 1046 raft_consensus.cc:3037] T 80471cfd4a944776baf05c4179f64a10 P f58850eb82214ad39880e45777bf51b4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:03.170696 1046 raft_consensus.cc:504] T 80471cfd4a944776baf05c4179f64a10 P f58850eb82214ad39880e45777bf51b4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "f58850eb82214ad39880e45777bf51b4" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 33951 } }
I20240502 21:26:03.171870 1046 leader_election.cc:288] T 80471cfd4a944776baf05c4179f64a10 P f58850eb82214ad39880e45777bf51b4 [CANDIDATE]: Term 2 election: Requested vote from peers f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66:39503)
I20240502 21:26:03.172380 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80471cfd4a944776baf05c4179f64a10" candidate_uuid: "f58850eb82214ad39880e45777bf51b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5"
I20240502 21:26:03.172623 32261 raft_consensus.cc:3032] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:26:03.172731 32261 raft_consensus.cc:729] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6f8b5e2067c4c23b0c89a8c471626f5, State: Running, Role: LEADER
I20240502 21:26:03.173027 32261 consensus_queue.cc:260] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "f58850eb82214ad39880e45777bf51b4" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 33951 } }
I20240502 21:26:03.173358 32261 raft_consensus.cc:3037] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:03.176388 32261 raft_consensus.cc:2454] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58850eb82214ad39880e45777bf51b4 in term 2.
I20240502 21:26:03.176806 977 leader_election.cc:302] T 80471cfd4a944776baf05c4179f64a10 P f58850eb82214ad39880e45777bf51b4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: f58850eb82214ad39880e45777bf51b4, f6f8b5e2067c4c23b0c89a8c471626f5; no voters:
I20240502 21:26:03.177217 2175 raft_consensus.cc:2781] T 80471cfd4a944776baf05c4179f64a10 P f58850eb82214ad39880e45777bf51b4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:26:03.177384 2175 raft_consensus.cc:686] T 80471cfd4a944776baf05c4179f64a10 P f58850eb82214ad39880e45777bf51b4 [term 2 LEADER]: Becoming Leader. State: Replica: f58850eb82214ad39880e45777bf51b4, State: Running, Role: LEADER
I20240502 21:26:03.177655 2175 consensus_queue.cc:237] T 80471cfd4a944776baf05c4179f64a10 P f58850eb82214ad39880e45777bf51b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "f58850eb82214ad39880e45777bf51b4" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 33951 } }
I20240502 21:26:03.177821 1157 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:03.178846 890 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:03.183146 1420 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:03.191988 31978 catalog_manager.cc:5484] T 80471cfd4a944776baf05c4179f64a10 P f58850eb82214ad39880e45777bf51b4 reported cstate change: term changed from 1 to 2, leader changed from f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66) to f58850eb82214ad39880e45777bf51b4 (127.29.82.68). New cstate: current_term: 2 leader_uuid: "f58850eb82214ad39880e45777bf51b4" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "f58850eb82214ad39880e45777bf51b4" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 33951 } } }
I20240502 21:26:03.192308 1288 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:03.203487 32107 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:26:03.210929 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:03.217104 32242 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:26:03.246491 32127 raft_consensus.cc:1261] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9 [term 2 FOLLOWER]: Refusing update from remote peer f58850eb82214ad39880e45777bf51b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 21:26:03.247339 2175 consensus_queue.cc:1035] T 9812cd5d31154807a4f605eb615553f1 P f58850eb82214ad39880e45777bf51b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:03.397328 1157 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:03.399562 1420 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:03.408321 890 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:03.421944 1288 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:03.460037 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:03.469766 32107 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:26:03.498811 32242 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:26:03.542968 32127 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d484c2aeb0904564997462a634b541d3"
dest_uuid: "cb8caceef23e475f8304b1ec1175e4b9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50242
I20240502 21:26:03.543207 32127 raft_consensus.cc:595] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:26:03.596740 32261 raft_consensus.cc:1261] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 2 FOLLOWER]: Refusing update from remote peer f58850eb82214ad39880e45777bf51b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:26:03.597353 2175 consensus_queue.cc:1035] T 80471cfd4a944776baf05c4179f64a10 P f58850eb82214ad39880e45777bf51b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:03.644073 1420 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:03.646152 1157 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:03.653993 890 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:03.662551 32107 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:26:03.665869 1288 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:03.675879 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:03.692207 32242 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:26:03.732908 2173 raft_consensus.cc:979] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9: : Instructing follower e705f0dda7904364a7453ee6294b3601 to start an election
I20240502 21:26:03.733070 2173 raft_consensus.cc:1067] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Signalling peer e705f0dda7904364a7453ee6294b3601 to start an election
I20240502 21:26:03.734107 1308 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d484c2aeb0904564997462a634b541d3"
dest_uuid: "e705f0dda7904364a7453ee6294b3601"
from {username='slave'} at 127.29.82.65:38879
I20240502 21:26:03.734354 1308 raft_consensus.cc:482] T d484c2aeb0904564997462a634b541d3 P e705f0dda7904364a7453ee6294b3601 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:26:03.734467 1308 raft_consensus.cc:3037] T d484c2aeb0904564997462a634b541d3 P e705f0dda7904364a7453ee6294b3601 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:03.737769 1308 raft_consensus.cc:504] T d484c2aeb0904564997462a634b541d3 P e705f0dda7904364a7453ee6294b3601 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "e705f0dda7904364a7453ee6294b3601" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 45931 } }
I20240502 21:26:03.738896 32127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d484c2aeb0904564997462a634b541d3" candidate_uuid: "e705f0dda7904364a7453ee6294b3601" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "cb8caceef23e475f8304b1ec1175e4b9"
I20240502 21:26:03.739140 32127 raft_consensus.cc:3032] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:26:03.739248 32127 raft_consensus.cc:729] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cb8caceef23e475f8304b1ec1175e4b9, State: Running, Role: LEADER
I20240502 21:26:03.739550 32127 consensus_queue.cc:260] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "e705f0dda7904364a7453ee6294b3601" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 45931 } }
I20240502 21:26:03.739789 32127 raft_consensus.cc:3037] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:03.742691 32127 raft_consensus.cc:2454] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e705f0dda7904364a7453ee6294b3601 in term 2.
I20240502 21:26:03.743327 1244 leader_election.cc:302] T d484c2aeb0904564997462a634b541d3 P e705f0dda7904364a7453ee6294b3601 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: cb8caceef23e475f8304b1ec1175e4b9, e705f0dda7904364a7453ee6294b3601; no voters:
I20240502 21:26:03.743696 2288 raft_consensus.cc:2781] T d484c2aeb0904564997462a634b541d3 P e705f0dda7904364a7453ee6294b3601 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:26:03.744153 2288 raft_consensus.cc:686] T d484c2aeb0904564997462a634b541d3 P e705f0dda7904364a7453ee6294b3601 [term 2 LEADER]: Becoming Leader. State: Replica: e705f0dda7904364a7453ee6294b3601, State: Running, Role: LEADER
I20240502 21:26:03.744513 2288 consensus_queue.cc:237] T d484c2aeb0904564997462a634b541d3 P e705f0dda7904364a7453ee6294b3601 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "e705f0dda7904364a7453ee6294b3601" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 45931 } }
I20240502 21:26:03.743700 1308 leader_election.cc:288] T d484c2aeb0904564997462a634b541d3 P e705f0dda7904364a7453ee6294b3601 [CANDIDATE]: Term 2 election: Requested vote from peers cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65:37947)
I20240502 21:26:03.747792 31971 catalog_manager.cc:5484] T d484c2aeb0904564997462a634b541d3 P e705f0dda7904364a7453ee6294b3601 reported cstate change: term changed from 1 to 2, leader changed from cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65) to e705f0dda7904364a7453ee6294b3601 (127.29.82.70). New cstate: current_term: 2 leader_uuid: "e705f0dda7904364a7453ee6294b3601" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "e705f0dda7904364a7453ee6294b3601" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 45931 } } }
I20240502 21:26:03.858387 890 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:03.863802 1157 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:03.872176 1420 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:03.874764 32107 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:26:03.887091 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:03.912600 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:03.931134 32242 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:26:03.996551 32127 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4919235c52834df5b3924d93391f171a"
dest_uuid: "cb8caceef23e475f8304b1ec1175e4b9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60820
I20240502 21:26:03.996773 32127 raft_consensus.cc:595] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:26:04.010787 2173 raft_consensus.cc:979] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9: : Instructing follower 98ac0bd5300e42c38bee7aae9634c7c9 to start an election
I20240502 21:26:04.011010 2173 raft_consensus.cc:1067] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Signalling peer 98ac0bd5300e42c38bee7aae9634c7c9 to start an election
I20240502 21:26:04.011905 1440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4919235c52834df5b3924d93391f171a"
dest_uuid: "98ac0bd5300e42c38bee7aae9634c7c9"
from {username='slave'} at 127.29.82.65:34389
I20240502 21:26:04.012219 1440 raft_consensus.cc:482] T 4919235c52834df5b3924d93391f171a P 98ac0bd5300e42c38bee7aae9634c7c9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:26:04.012428 1440 raft_consensus.cc:3037] T 4919235c52834df5b3924d93391f171a P 98ac0bd5300e42c38bee7aae9634c7c9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:04.015542 1440 raft_consensus.cc:504] T 4919235c52834df5b3924d93391f171a P 98ac0bd5300e42c38bee7aae9634c7c9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "98ac0bd5300e42c38bee7aae9634c7c9" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 39493 } }
I20240502 21:26:04.017180 1440 leader_election.cc:288] T 4919235c52834df5b3924d93391f171a P 98ac0bd5300e42c38bee7aae9634c7c9 [CANDIDATE]: Term 2 election: Requested vote from peers cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65:37947)
I20240502 21:26:04.017678 32127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4919235c52834df5b3924d93391f171a" candidate_uuid: "98ac0bd5300e42c38bee7aae9634c7c9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "cb8caceef23e475f8304b1ec1175e4b9"
I20240502 21:26:04.017930 32127 raft_consensus.cc:3032] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:26:04.018039 32127 raft_consensus.cc:729] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cb8caceef23e475f8304b1ec1175e4b9, State: Running, Role: LEADER
I20240502 21:26:04.018311 32127 consensus_queue.cc:260] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "98ac0bd5300e42c38bee7aae9634c7c9" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 39493 } }
I20240502 21:26:04.018606 32127 raft_consensus.cc:3037] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:04.021441 32127 raft_consensus.cc:2454] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98ac0bd5300e42c38bee7aae9634c7c9 in term 2.
I20240502 21:26:04.021961 1375 leader_election.cc:302] T 4919235c52834df5b3924d93391f171a P 98ac0bd5300e42c38bee7aae9634c7c9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 98ac0bd5300e42c38bee7aae9634c7c9, cb8caceef23e475f8304b1ec1175e4b9; no voters:
I20240502 21:26:04.022747 2388 raft_consensus.cc:2781] T 4919235c52834df5b3924d93391f171a P 98ac0bd5300e42c38bee7aae9634c7c9 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:26:04.023262 2388 raft_consensus.cc:686] T 4919235c52834df5b3924d93391f171a P 98ac0bd5300e42c38bee7aae9634c7c9 [term 2 LEADER]: Becoming Leader. State: Replica: 98ac0bd5300e42c38bee7aae9634c7c9, State: Running, Role: LEADER
I20240502 21:26:04.023926 2388 consensus_queue.cc:237] T 4919235c52834df5b3924d93391f171a P 98ac0bd5300e42c38bee7aae9634c7c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "98ac0bd5300e42c38bee7aae9634c7c9" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 39493 } }
I20240502 21:26:04.027871 31978 catalog_manager.cc:5484] T 4919235c52834df5b3924d93391f171a P 98ac0bd5300e42c38bee7aae9634c7c9 reported cstate change: term changed from 1 to 2, leader changed from cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65) to 98ac0bd5300e42c38bee7aae9634c7c9 (127.29.82.71). New cstate: current_term: 2 leader_uuid: "98ac0bd5300e42c38bee7aae9634c7c9" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "98ac0bd5300e42c38bee7aae9634c7c9" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 39493 } } }
I20240502 21:26:04.094317 1157 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:04.100111 1420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:04.103114 890 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:04.110013 32107 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:26:04.127444 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:04.154088 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:04.155541 32242 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:26:04.168479 32127 raft_consensus.cc:1261] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9 [term 2 FOLLOWER]: Refusing update from remote peer e705f0dda7904364a7453ee6294b3601: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:26:04.169245 2288 consensus_queue.cc:1035] T d484c2aeb0904564997462a634b541d3 P e705f0dda7904364a7453ee6294b3601 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:04.314800 1157 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:04.316680 890 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:04.326843 32107 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:26:04.329212 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:04.339915 1420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:04.343889 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:04.372244 32242 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:26:04.442489 32126 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "699c3a3f730248bdacf122126883f2f4"
dest_uuid: "cb8caceef23e475f8304b1ec1175e4b9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60870
I20240502 21:26:04.442728 32126 raft_consensus.cc:595] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:26:04.494709 2333 raft_consensus.cc:979] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9: : Instructing follower b399bf9ec0c544749c7eead22ddfedf6 to start an election
I20240502 21:26:04.494910 2333 raft_consensus.cc:1067] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Signalling peer b399bf9ec0c544749c7eead22ddfedf6 to start an election
I20240502 21:26:04.495695 911 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "699c3a3f730248bdacf122126883f2f4"
dest_uuid: "b399bf9ec0c544749c7eead22ddfedf6"
from {username='slave'} at 127.29.82.65:54611
I20240502 21:26:04.495954 911 raft_consensus.cc:482] T 699c3a3f730248bdacf122126883f2f4 P b399bf9ec0c544749c7eead22ddfedf6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:26:04.496094 911 raft_consensus.cc:3037] T 699c3a3f730248bdacf122126883f2f4 P b399bf9ec0c544749c7eead22ddfedf6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:04.499233 911 raft_consensus.cc:504] T 699c3a3f730248bdacf122126883f2f4 P b399bf9ec0c544749c7eead22ddfedf6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "b399bf9ec0c544749c7eead22ddfedf6" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 45329 } }
I20240502 21:26:04.499852 911 leader_election.cc:288] T 699c3a3f730248bdacf122126883f2f4 P b399bf9ec0c544749c7eead22ddfedf6 [CANDIDATE]: Term 2 election: Requested vote from peers cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65:37947)
I20240502 21:26:04.500591 32126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699c3a3f730248bdacf122126883f2f4" candidate_uuid: "b399bf9ec0c544749c7eead22ddfedf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "cb8caceef23e475f8304b1ec1175e4b9"
I20240502 21:26:04.500810 32126 raft_consensus.cc:3032] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:26:04.500921 32126 raft_consensus.cc:729] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cb8caceef23e475f8304b1ec1175e4b9, State: Running, Role: LEADER
I20240502 21:26:04.501267 32126 consensus_queue.cc:260] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "b399bf9ec0c544749c7eead22ddfedf6" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 45329 } }
I20240502 21:26:04.501623 32126 raft_consensus.cc:3037] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:04.503952 32126 raft_consensus.cc:2454] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b399bf9ec0c544749c7eead22ddfedf6 in term 2.
I20240502 21:26:04.504416 846 leader_election.cc:302] T 699c3a3f730248bdacf122126883f2f4 P b399bf9ec0c544749c7eead22ddfedf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b399bf9ec0c544749c7eead22ddfedf6, cb8caceef23e475f8304b1ec1175e4b9; no voters:
I20240502 21:26:04.504793 2081 raft_consensus.cc:2781] T 699c3a3f730248bdacf122126883f2f4 P b399bf9ec0c544749c7eead22ddfedf6 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:26:04.516485 2081 raft_consensus.cc:686] T 699c3a3f730248bdacf122126883f2f4 P b399bf9ec0c544749c7eead22ddfedf6 [term 2 LEADER]: Becoming Leader. State: Replica: b399bf9ec0c544749c7eead22ddfedf6, State: Running, Role: LEADER
I20240502 21:26:04.516932 2081 consensus_queue.cc:237] T 699c3a3f730248bdacf122126883f2f4 P b399bf9ec0c544749c7eead22ddfedf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "b399bf9ec0c544749c7eead22ddfedf6" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 45329 } }
I20240502 21:26:04.521636 31975 catalog_manager.cc:5484] T 699c3a3f730248bdacf122126883f2f4 P b399bf9ec0c544749c7eead22ddfedf6 reported cstate change: term changed from 1 to 2, leader changed from cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65) to b399bf9ec0c544749c7eead22ddfedf6 (127.29.82.67). New cstate: current_term: 2 leader_uuid: "b399bf9ec0c544749c7eead22ddfedf6" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "b399bf9ec0c544749c7eead22ddfedf6" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 45329 } } }
I20240502 21:26:04.545321 1157 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:04.548226 1420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:04.565049 32107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:04.574069 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:04.585919 32126 raft_consensus.cc:1261] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9 [term 2 FOLLOWER]: Refusing update from remote peer 98ac0bd5300e42c38bee7aae9634c7c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:26:04.600536 2492 consensus_queue.cc:1035] T 4919235c52834df5b3924d93391f171a P 98ac0bd5300e42c38bee7aae9634c7c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20240502 21:26:04.604748 32242 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:26:04.609822 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:04.614845 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:04.785459 1157 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:04.795514 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:04.797207 1420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:04.807127 32107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:04.807809 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:04.827119 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:04.840819 32242 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:26:04.907191 32261 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3843ac68594f43079a1253a877c48065"
dest_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52936
I20240502 21:26:04.907406 32261 raft_consensus.cc:595] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:26:04.999783 1157 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:05.011613 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:05.013623 1420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:05.014102 32107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:05.019064 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:05.033797 32126 raft_consensus.cc:1261] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9 [term 2 FOLLOWER]: Refusing update from remote peer b399bf9ec0c544749c7eead22ddfedf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:26:05.034791 2081 consensus_queue.cc:1035] T 699c3a3f730248bdacf122126883f2f4 P b399bf9ec0c544749c7eead22ddfedf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:05.037441 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:05.039669 32242 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:26:05.066016 2573 raft_consensus.cc:979] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5: : Instructing follower 21e73dc1a0eb486db9717fa320347e1c to start an election
I20240502 21:26:05.066300 2249 raft_consensus.cc:1067] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Signalling peer 21e73dc1a0eb486db9717fa320347e1c to start an election
I20240502 21:26:05.067797 1177 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3843ac68594f43079a1253a877c48065"
dest_uuid: "21e73dc1a0eb486db9717fa320347e1c"
from {username='slave'} at 127.29.82.66:50035
I20240502 21:26:05.068142 1177 raft_consensus.cc:482] T 3843ac68594f43079a1253a877c48065 P 21e73dc1a0eb486db9717fa320347e1c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:26:05.068356 1177 raft_consensus.cc:3037] T 3843ac68594f43079a1253a877c48065 P 21e73dc1a0eb486db9717fa320347e1c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:05.071529 1177 raft_consensus.cc:504] T 3843ac68594f43079a1253a877c48065 P 21e73dc1a0eb486db9717fa320347e1c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "21e73dc1a0eb486db9717fa320347e1c" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 36679 } }
I20240502 21:26:05.114898 1177 leader_election.cc:288] T 3843ac68594f43079a1253a877c48065 P 21e73dc1a0eb486db9717fa320347e1c [CANDIDATE]: Term 2 election: Requested vote from peers f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66:39503)
I20240502 21:26:05.115434 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3843ac68594f43079a1253a877c48065" candidate_uuid: "21e73dc1a0eb486db9717fa320347e1c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5"
I20240502 21:26:05.115702 32261 raft_consensus.cc:3032] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:26:05.115784 32261 raft_consensus.cc:729] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6f8b5e2067c4c23b0c89a8c471626f5, State: Running, Role: LEADER
I20240502 21:26:05.116065 32261 consensus_queue.cc:260] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "21e73dc1a0eb486db9717fa320347e1c" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 36679 } }
I20240502 21:26:05.116336 32261 raft_consensus.cc:3037] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:05.118234 32261 raft_consensus.cc:2454] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21e73dc1a0eb486db9717fa320347e1c in term 2.
I20240502 21:26:05.119067 1110 leader_election.cc:302] T 3843ac68594f43079a1253a877c48065 P 21e73dc1a0eb486db9717fa320347e1c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 21e73dc1a0eb486db9717fa320347e1c, f6f8b5e2067c4c23b0c89a8c471626f5; no voters:
I20240502 21:26:05.124681 2575 raft_consensus.cc:2781] T 3843ac68594f43079a1253a877c48065 P 21e73dc1a0eb486db9717fa320347e1c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:26:05.131914 2575 raft_consensus.cc:686] T 3843ac68594f43079a1253a877c48065 P 21e73dc1a0eb486db9717fa320347e1c [term 2 LEADER]: Becoming Leader. State: Replica: 21e73dc1a0eb486db9717fa320347e1c, State: Running, Role: LEADER
I20240502 21:26:05.132632 2575 consensus_queue.cc:237] T 3843ac68594f43079a1253a877c48065 P 21e73dc1a0eb486db9717fa320347e1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "21e73dc1a0eb486db9717fa320347e1c" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 36679 } }
I20240502 21:26:05.136898 31971 catalog_manager.cc:5484] T 3843ac68594f43079a1253a877c48065 P 21e73dc1a0eb486db9717fa320347e1c reported cstate change: term changed from 1 to 2, leader changed from f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66) to 21e73dc1a0eb486db9717fa320347e1c (127.29.82.69). New cstate: current_term: 2 leader_uuid: "21e73dc1a0eb486db9717fa320347e1c" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "21e73dc1a0eb486db9717fa320347e1c" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 36679 } } }
I20240502 21:26:05.267560 1157 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:05.279665 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:05.287312 1420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:05.297955 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:05.304476 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:05.304817 32107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:05.316133 32242 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:26:05.394120 32261 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a80b4f90d4b04e1886e494cf23a6284b"
dest_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52958
I20240502 21:26:05.394327 32261 raft_consensus.cc:595] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:26:05.484680 1157 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:05.496184 1420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:05.502028 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:05.504503 32107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:05.513092 32242 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:26:05.513545 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:05.539917 32261 raft_consensus.cc:1261] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 2 FOLLOWER]: Refusing update from remote peer 21e73dc1a0eb486db9717fa320347e1c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:26:05.540822 2575 consensus_queue.cc:1035] T 3843ac68594f43079a1253a877c48065 P 21e73dc1a0eb486db9717fa320347e1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:05.542930 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:05.715788 1157 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:05.722584 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:05.730165 32107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:05.731171 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:05.739053 1420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:05.740279 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:05.741133 32242 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:26:05.785851 2711 raft_consensus.cc:979] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5: : Instructing follower 21e73dc1a0eb486db9717fa320347e1c to start an election
I20240502 21:26:05.787217 2573 raft_consensus.cc:1067] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Signalling peer 21e73dc1a0eb486db9717fa320347e1c to start an election
I20240502 21:26:05.788096 1177 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a80b4f90d4b04e1886e494cf23a6284b"
dest_uuid: "21e73dc1a0eb486db9717fa320347e1c"
from {username='slave'} at 127.29.82.66:50035
I20240502 21:26:05.788378 1177 raft_consensus.cc:482] T a80b4f90d4b04e1886e494cf23a6284b P 21e73dc1a0eb486db9717fa320347e1c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:26:05.788508 1177 raft_consensus.cc:3037] T a80b4f90d4b04e1886e494cf23a6284b P 21e73dc1a0eb486db9717fa320347e1c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:05.791795 1177 raft_consensus.cc:504] T a80b4f90d4b04e1886e494cf23a6284b P 21e73dc1a0eb486db9717fa320347e1c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "21e73dc1a0eb486db9717fa320347e1c" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 36679 } }
I20240502 21:26:05.793051 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80b4f90d4b04e1886e494cf23a6284b" candidate_uuid: "21e73dc1a0eb486db9717fa320347e1c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5"
I20240502 21:26:05.793633 1177 leader_election.cc:288] T a80b4f90d4b04e1886e494cf23a6284b P 21e73dc1a0eb486db9717fa320347e1c [CANDIDATE]: Term 2 election: Requested vote from peers f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66:39503)
I20240502 21:26:05.794255 32263 raft_consensus.cc:3032] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:26:05.794376 32263 raft_consensus.cc:729] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6f8b5e2067c4c23b0c89a8c471626f5, State: Running, Role: LEADER
I20240502 21:26:05.794662 32263 consensus_queue.cc:260] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "21e73dc1a0eb486db9717fa320347e1c" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 36679 } }
I20240502 21:26:05.795019 32263 raft_consensus.cc:3037] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:05.798198 32263 raft_consensus.cc:2454] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21e73dc1a0eb486db9717fa320347e1c in term 2.
I20240502 21:26:05.798897 1110 leader_election.cc:302] T a80b4f90d4b04e1886e494cf23a6284b P 21e73dc1a0eb486db9717fa320347e1c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 21e73dc1a0eb486db9717fa320347e1c, f6f8b5e2067c4c23b0c89a8c471626f5; no voters:
I20240502 21:26:05.799335 2575 raft_consensus.cc:2781] T a80b4f90d4b04e1886e494cf23a6284b P 21e73dc1a0eb486db9717fa320347e1c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:26:05.799465 2575 raft_consensus.cc:686] T a80b4f90d4b04e1886e494cf23a6284b P 21e73dc1a0eb486db9717fa320347e1c [term 2 LEADER]: Becoming Leader. State: Replica: 21e73dc1a0eb486db9717fa320347e1c, State: Running, Role: LEADER
I20240502 21:26:05.799958 2575 consensus_queue.cc:237] T a80b4f90d4b04e1886e494cf23a6284b P 21e73dc1a0eb486db9717fa320347e1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "21e73dc1a0eb486db9717fa320347e1c" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 36679 } }
I20240502 21:26:05.807885 31975 catalog_manager.cc:5484] T a80b4f90d4b04e1886e494cf23a6284b P 21e73dc1a0eb486db9717fa320347e1c reported cstate change: term changed from 1 to 2, leader changed from f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66) to 21e73dc1a0eb486db9717fa320347e1c (127.29.82.69). New cstate: current_term: 2 leader_uuid: "21e73dc1a0eb486db9717fa320347e1c" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "21e73dc1a0eb486db9717fa320347e1c" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 36679 } } }
I20240502 21:26:05.845942 32263 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cd16f492f22c478a946d1da05f6a91b4"
dest_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52998
I20240502 21:26:05.846143 32263 raft_consensus.cc:595] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:26:05.947597 2711 raft_consensus.cc:979] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5: : Instructing follower 98ac0bd5300e42c38bee7aae9634c7c9 to start an election
I20240502 21:26:05.947796 2711 raft_consensus.cc:1067] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Signalling peer 98ac0bd5300e42c38bee7aae9634c7c9 to start an election
I20240502 21:26:05.949466 1440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cd16f492f22c478a946d1da05f6a91b4"
dest_uuid: "98ac0bd5300e42c38bee7aae9634c7c9"
from {username='slave'} at 127.29.82.66:43399
I20240502 21:26:05.949772 1440 raft_consensus.cc:482] T cd16f492f22c478a946d1da05f6a91b4 P 98ac0bd5300e42c38bee7aae9634c7c9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:26:05.949899 1440 raft_consensus.cc:3037] T cd16f492f22c478a946d1da05f6a91b4 P 98ac0bd5300e42c38bee7aae9634c7c9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:05.950464 1157 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:05.952611 1420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:05.953491 1440 raft_consensus.cc:504] T cd16f492f22c478a946d1da05f6a91b4 P 98ac0bd5300e42c38bee7aae9634c7c9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "98ac0bd5300e42c38bee7aae9634c7c9" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 39493 } }
I20240502 21:26:05.954236 1440 leader_election.cc:288] T cd16f492f22c478a946d1da05f6a91b4 P 98ac0bd5300e42c38bee7aae9634c7c9 [CANDIDATE]: Term 2 election: Requested vote from peers f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66:39503)
I20240502 21:26:05.955003 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd16f492f22c478a946d1da05f6a91b4" candidate_uuid: "98ac0bd5300e42c38bee7aae9634c7c9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5"
I20240502 21:26:05.955255 32263 raft_consensus.cc:3032] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:26:05.955371 32263 raft_consensus.cc:729] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6f8b5e2067c4c23b0c89a8c471626f5, State: Running, Role: LEADER
I20240502 21:26:05.956028 32263 consensus_queue.cc:260] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "98ac0bd5300e42c38bee7aae9634c7c9" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 39493 } }
I20240502 21:26:05.956339 32263 raft_consensus.cc:3037] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:05.956458 32107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:05.967154 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:05.967767 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:05.972527 32263 raft_consensus.cc:2454] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98ac0bd5300e42c38bee7aae9634c7c9 in term 2.
I20240502 21:26:05.980103 1372 leader_election.cc:302] T cd16f492f22c478a946d1da05f6a91b4 P 98ac0bd5300e42c38bee7aae9634c7c9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 98ac0bd5300e42c38bee7aae9634c7c9, f6f8b5e2067c4c23b0c89a8c471626f5; no voters:
I20240502 21:26:05.980594 2699 raft_consensus.cc:2781] T cd16f492f22c478a946d1da05f6a91b4 P 98ac0bd5300e42c38bee7aae9634c7c9 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:26:05.981640 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:06.003605 2699 raft_consensus.cc:686] T cd16f492f22c478a946d1da05f6a91b4 P 98ac0bd5300e42c38bee7aae9634c7c9 [term 2 LEADER]: Becoming Leader. State: Replica: 98ac0bd5300e42c38bee7aae9634c7c9, State: Running, Role: LEADER
I20240502 21:26:06.004068 2699 consensus_queue.cc:237] T cd16f492f22c478a946d1da05f6a91b4 P 98ac0bd5300e42c38bee7aae9634c7c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "98ac0bd5300e42c38bee7aae9634c7c9" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 39493 } }
I20240502 21:26:06.006991 31971 catalog_manager.cc:5484] T cd16f492f22c478a946d1da05f6a91b4 P 98ac0bd5300e42c38bee7aae9634c7c9 reported cstate change: term changed from 1 to 2, leader changed from f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66) to 98ac0bd5300e42c38bee7aae9634c7c9 (127.29.82.71). New cstate: current_term: 2 leader_uuid: "98ac0bd5300e42c38bee7aae9634c7c9" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "98ac0bd5300e42c38bee7aae9634c7c9" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 39493 } } }
I20240502 21:26:06.024524 32242 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:06.368075 1157 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:06.371193 32263 raft_consensus.cc:1261] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5 [term 2 FOLLOWER]: Refusing update from remote peer 21e73dc1a0eb486db9717fa320347e1c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:26:06.371918 2575 consensus_queue.cc:1035] T a80b4f90d4b04e1886e494cf23a6284b P 21e73dc1a0eb486db9717fa320347e1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:06.373100 1420 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:06.379997 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:06.411789 32107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:06.435842 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:06.469733 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:06.475992 32242 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:06.554018 1308 consensus_queue.cc:237] T 9873952de1224793b8b7f17ef92295b7 P e705f0dda7904364a7453ee6294b3601 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e705f0dda7904364a7453ee6294b3601" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 45931 } }
I20240502 21:26:06.555523 2288 raft_consensus.cc:2932] T 9873952de1224793b8b7f17ef92295b7 P e705f0dda7904364a7453ee6294b3601 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e705f0dda7904364a7453ee6294b3601" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 45931 } } }
I20240502 21:26:06.562117 31974 catalog_manager.cc:5484] T 9873952de1224793b8b7f17ef92295b7 P e705f0dda7904364a7453ee6294b3601 reported cstate change: config changed from index 4 to 6, VOTER cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65) evicted. New cstate: current_term: 2 leader_uuid: "e705f0dda7904364a7453ee6294b3601" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e705f0dda7904364a7453ee6294b3601" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 45931 } } }
I20240502 21:26:06.567243 32107 tablet_service.cc:1514] Processing DeleteTablet for tablet 9873952de1224793b8b7f17ef92295b7 with delete_type TABLET_DATA_TOMBSTONED (TS cb8caceef23e475f8304b1ec1175e4b9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35662
I20240502 21:26:06.568221 2821 tablet_replica.cc:330] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9: stopping tablet replica
I20240502 21:26:06.568727 2821 raft_consensus.cc:2229] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:06.569056 2821 raft_consensus.cc:2258] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:06.570639 2821 ts_tablet_manager.cc:1875] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:06.576978 2821 ts_tablet_manager.cc:1888] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 21:26:06.577132 2821 log.cc:1196] T 9873952de1224793b8b7f17ef92295b7 P cb8caceef23e475f8304b1ec1175e4b9: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/9873952de1224793b8b7f17ef92295b7
I20240502 21:26:06.577984 31967 catalog_manager.cc:4830] TS cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65:37947): tablet 9873952de1224793b8b7f17ef92295b7 (table rebalance_test_table_3 [id=74a97a8ddca948248cc5d22c9b96616c]) successfully deleted
I20240502 21:26:06.616881 32263 raft_consensus.cc:1261] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 2 FOLLOWER]: Refusing update from remote peer 98ac0bd5300e42c38bee7aae9634c7c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:26:06.617528 2699 consensus_queue.cc:1035] T cd16f492f22c478a946d1da05f6a91b4 P 98ac0bd5300e42c38bee7aae9634c7c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:06.660295 1157 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:06.662326 1420 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:06.682029 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:06.685110 32107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:06.696651 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:06.703933 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:06.711686 32242 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:06.811205 911 consensus_queue.cc:237] T 43d4ba4bfe70416bb9c06a3bfe299835 P b399bf9ec0c544749c7eead22ddfedf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b399bf9ec0c544749c7eead22ddfedf6" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 45329 } }
I20240502 21:26:06.813524 2574 raft_consensus.cc:2932] T 43d4ba4bfe70416bb9c06a3bfe299835 P b399bf9ec0c544749c7eead22ddfedf6 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b399bf9ec0c544749c7eead22ddfedf6" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 45329 } } }
I20240502 21:26:06.821173 31975 catalog_manager.cc:5484] T 43d4ba4bfe70416bb9c06a3bfe299835 P b399bf9ec0c544749c7eead22ddfedf6 reported cstate change: config changed from index 4 to 6, VOTER f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66) evicted. New cstate: current_term: 2 leader_uuid: "b399bf9ec0c544749c7eead22ddfedf6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b399bf9ec0c544749c7eead22ddfedf6" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 45329 } } }
I20240502 21:26:06.825371 32242 tablet_service.cc:1514] Processing DeleteTablet for tablet 43d4ba4bfe70416bb9c06a3bfe299835 with delete_type TABLET_DATA_TOMBSTONED (TS f6f8b5e2067c4c23b0c89a8c471626f5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46274
I20240502 21:26:06.826458 2878 tablet_replica.cc:330] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5: stopping tablet replica
I20240502 21:26:06.826943 2878 raft_consensus.cc:2229] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:06.827257 2878 raft_consensus.cc:2258] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:06.828536 2878 ts_tablet_manager.cc:1875] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:06.834487 2878 ts_tablet_manager.cc:1888] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 21:26:06.834694 2878 log.cc:1196] T 43d4ba4bfe70416bb9c06a3bfe299835 P f6f8b5e2067c4c23b0c89a8c471626f5: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/43d4ba4bfe70416bb9c06a3bfe299835
I20240502 21:26:06.835711 31964 catalog_manager.cc:4830] TS f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66:39503): tablet 43d4ba4bfe70416bb9c06a3bfe299835 (table rebalance_test_table_2 [id=bff79bbad6b54d32b0ee9781a39e2579]) successfully deleted
I20240502 21:26:06.932452 1157 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:06.937261 1420 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:06.953842 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:06.956714 32107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:06.983492 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:07.016827 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:07.018842 32242 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:07.115327 1046 consensus_queue.cc:237] T 9812cd5d31154807a4f605eb615553f1 P f58850eb82214ad39880e45777bf51b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f58850eb82214ad39880e45777bf51b4" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 33951 } }
I20240502 21:26:07.117194 2176 raft_consensus.cc:2932] T 9812cd5d31154807a4f605eb615553f1 P f58850eb82214ad39880e45777bf51b4 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65) evicted. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f58850eb82214ad39880e45777bf51b4" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 33951 } } }
I20240502 21:26:07.125038 31980 catalog_manager.cc:5484] T 9812cd5d31154807a4f605eb615553f1 P f58850eb82214ad39880e45777bf51b4 reported cstate change: config changed from index 2 to 4, VOTER cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65) evicted. New cstate: current_term: 2 leader_uuid: "f58850eb82214ad39880e45777bf51b4" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f58850eb82214ad39880e45777bf51b4" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 33951 } } }
I20240502 21:26:07.129603 32107 tablet_service.cc:1514] Processing DeleteTablet for tablet 9812cd5d31154807a4f605eb615553f1 with delete_type TABLET_DATA_TOMBSTONED (TS cb8caceef23e475f8304b1ec1175e4b9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35662
I20240502 21:26:07.141820 2930 tablet_replica.cc:330] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9: stopping tablet replica
I20240502 21:26:07.142107 2930 raft_consensus.cc:2229] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:07.142346 2930 raft_consensus.cc:2258] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:07.143182 2930 ts_tablet_manager.cc:1875] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:07.151743 2930 ts_tablet_manager.cc:1888] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240502 21:26:07.151952 2930 log.cc:1196] T 9812cd5d31154807a4f605eb615553f1 P cb8caceef23e475f8304b1ec1175e4b9: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/9812cd5d31154807a4f605eb615553f1
I20240502 21:26:07.152927 31967 catalog_manager.cc:4830] TS cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65:37947): tablet 9812cd5d31154807a4f605eb615553f1 (table TestTable [id=ebf8d845a721492e8b62cd5dc012abae]) successfully deleted
I20240502 21:26:07.256096 1157 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:07.256527 1420 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:07.279963 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:07.303748 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:07.338863 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:07.340021 32107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:07.367813 32242 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:07.476635 1046 consensus_queue.cc:237] T 80471cfd4a944776baf05c4179f64a10 P f58850eb82214ad39880e45777bf51b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f58850eb82214ad39880e45777bf51b4" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 33951 } }
I20240502 21:26:07.477905 2176 raft_consensus.cc:2932] T 80471cfd4a944776baf05c4179f64a10 P f58850eb82214ad39880e45777bf51b4 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f58850eb82214ad39880e45777bf51b4" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 33951 } } }
I20240502 21:26:07.481981 31971 catalog_manager.cc:5484] T 80471cfd4a944776baf05c4179f64a10 P f58850eb82214ad39880e45777bf51b4 reported cstate change: config changed from index 4 to 6, VOTER f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66) evicted. New cstate: current_term: 2 leader_uuid: "f58850eb82214ad39880e45777bf51b4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f58850eb82214ad39880e45777bf51b4" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 33951 } } }
I20240502 21:26:07.486079 32242 tablet_service.cc:1514] Processing DeleteTablet for tablet 80471cfd4a944776baf05c4179f64a10 with delete_type TABLET_DATA_TOMBSTONED (TS f6f8b5e2067c4c23b0c89a8c471626f5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46274
I20240502 21:26:07.486806 2990 tablet_replica.cc:330] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5: stopping tablet replica
I20240502 21:26:07.487093 2990 raft_consensus.cc:2229] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:07.487313 2990 raft_consensus.cc:2258] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:07.488008 2990 ts_tablet_manager.cc:1875] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:07.493121 2990 ts_tablet_manager.cc:1888] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 21:26:07.493245 2990 log.cc:1196] T 80471cfd4a944776baf05c4179f64a10 P f6f8b5e2067c4c23b0c89a8c471626f5: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/80471cfd4a944776baf05c4179f64a10
I20240502 21:26:07.493956 31964 catalog_manager.cc:4830] TS f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66:39503): tablet 80471cfd4a944776baf05c4179f64a10 (table rebalance_test_table_4 [id=0e0c0c30a67c4461bafc601a7c3a4676]) successfully deleted
I20240502 21:26:07.586298 1157 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:07.597177 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:07.597919 1420 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:07.615410 32107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:07.657949 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:07.658182 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:07.676796 32242 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:07.745838 1308 consensus_queue.cc:237] T d484c2aeb0904564997462a634b541d3 P e705f0dda7904364a7453ee6294b3601 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e705f0dda7904364a7453ee6294b3601" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 45931 } }
I20240502 21:26:07.747174 2288 raft_consensus.cc:2932] T d484c2aeb0904564997462a634b541d3 P e705f0dda7904364a7453ee6294b3601 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e705f0dda7904364a7453ee6294b3601" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 45931 } } }
I20240502 21:26:07.751801 31971 catalog_manager.cc:5484] T d484c2aeb0904564997462a634b541d3 P e705f0dda7904364a7453ee6294b3601 reported cstate change: config changed from index 4 to 6, VOTER cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65) evicted. New cstate: current_term: 2 leader_uuid: "e705f0dda7904364a7453ee6294b3601" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e705f0dda7904364a7453ee6294b3601" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 45931 } } }
I20240502 21:26:07.755864 32107 tablet_service.cc:1514] Processing DeleteTablet for tablet d484c2aeb0904564997462a634b541d3 with delete_type TABLET_DATA_TOMBSTONED (TS cb8caceef23e475f8304b1ec1175e4b9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35662
I20240502 21:26:07.756546 3043 tablet_replica.cc:330] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9: stopping tablet replica
I20240502 21:26:07.756753 3043 raft_consensus.cc:2229] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:07.756992 3043 raft_consensus.cc:2258] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:07.757714 3043 ts_tablet_manager.cc:1875] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:07.763978 3043 ts_tablet_manager.cc:1888] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 21:26:07.764194 3043 log.cc:1196] T d484c2aeb0904564997462a634b541d3 P cb8caceef23e475f8304b1ec1175e4b9: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/d484c2aeb0904564997462a634b541d3
I20240502 21:26:07.764896 31967 catalog_manager.cc:4830] TS cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65:37947): tablet d484c2aeb0904564997462a634b541d3 (table rebalance_test_table_1 [id=4a48efbb86944c3897b93b4662c377ae]) successfully deleted
I20240502 21:26:07.870389 1157 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:07.870486 1420 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:07.876206 32107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:07.880128 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:07.892069 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:07.906775 32242 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:07.927328 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:08.010219 1440 consensus_queue.cc:237] T 4919235c52834df5b3924d93391f171a P 98ac0bd5300e42c38bee7aae9634c7c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "98ac0bd5300e42c38bee7aae9634c7c9" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 39493 } }
I20240502 21:26:08.011551 3025 raft_consensus.cc:2932] T 4919235c52834df5b3924d93391f171a P 98ac0bd5300e42c38bee7aae9634c7c9 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "98ac0bd5300e42c38bee7aae9634c7c9" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 39493 } } }
I20240502 21:26:08.016664 31974 catalog_manager.cc:5484] T 4919235c52834df5b3924d93391f171a P 98ac0bd5300e42c38bee7aae9634c7c9 reported cstate change: config changed from index 4 to 6, VOTER cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65) evicted. New cstate: current_term: 2 leader_uuid: "98ac0bd5300e42c38bee7aae9634c7c9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "98ac0bd5300e42c38bee7aae9634c7c9" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 39493 } } }
I20240502 21:26:08.020543 32107 tablet_service.cc:1514] Processing DeleteTablet for tablet 4919235c52834df5b3924d93391f171a with delete_type TABLET_DATA_TOMBSTONED (TS cb8caceef23e475f8304b1ec1175e4b9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35662
I20240502 21:26:08.020915 3043 tablet_replica.cc:330] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9: stopping tablet replica
I20240502 21:26:08.021119 3043 raft_consensus.cc:2229] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:08.021327 3043 raft_consensus.cc:2258] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:08.021939 3043 ts_tablet_manager.cc:1875] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:08.026439 3043 ts_tablet_manager.cc:1888] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 21:26:08.026588 3043 log.cc:1196] T 4919235c52834df5b3924d93391f171a P cb8caceef23e475f8304b1ec1175e4b9: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/4919235c52834df5b3924d93391f171a
I20240502 21:26:08.027326 31967 catalog_manager.cc:4830] TS cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65:37947): tablet 4919235c52834df5b3924d93391f171a (table rebalance_test_table_3 [id=74a97a8ddca948248cc5d22c9b96616c]) successfully deleted
I20240502 21:26:08.133545 1157 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:08.138926 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:08.139751 32107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:08.140050 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:08.140204 1420 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:08.164438 32242 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:08.164897 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:08.251821 911 consensus_queue.cc:237] T 699c3a3f730248bdacf122126883f2f4 P b399bf9ec0c544749c7eead22ddfedf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b399bf9ec0c544749c7eead22ddfedf6" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 45329 } }
I20240502 21:26:08.253059 3103 raft_consensus.cc:2932] T 699c3a3f730248bdacf122126883f2f4 P b399bf9ec0c544749c7eead22ddfedf6 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b399bf9ec0c544749c7eead22ddfedf6" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 45329 } } }
I20240502 21:26:08.257493 31978 catalog_manager.cc:5484] T 699c3a3f730248bdacf122126883f2f4 P b399bf9ec0c544749c7eead22ddfedf6 reported cstate change: config changed from index 4 to 6, VOTER cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65) evicted. New cstate: current_term: 2 leader_uuid: "b399bf9ec0c544749c7eead22ddfedf6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b399bf9ec0c544749c7eead22ddfedf6" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 45329 } } }
I20240502 21:26:08.261071 32107 tablet_service.cc:1514] Processing DeleteTablet for tablet 699c3a3f730248bdacf122126883f2f4 with delete_type TABLET_DATA_TOMBSTONED (TS cb8caceef23e475f8304b1ec1175e4b9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35662
I20240502 21:26:08.261432 3043 tablet_replica.cc:330] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9: stopping tablet replica
I20240502 21:26:08.261627 3043 raft_consensus.cc:2229] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:08.261821 3043 raft_consensus.cc:2258] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:08.262552 3043 ts_tablet_manager.cc:1875] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:08.268366 3043 ts_tablet_manager.cc:1888] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 21:26:08.268468 3043 log.cc:1196] T 699c3a3f730248bdacf122126883f2f4 P cb8caceef23e475f8304b1ec1175e4b9: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/699c3a3f730248bdacf122126883f2f4
I20240502 21:26:08.269156 31967 catalog_manager.cc:4830] TS cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65:37947): tablet 699c3a3f730248bdacf122126883f2f4 (table rebalance_test_table_1 [id=4a48efbb86944c3897b93b4662c377ae]) successfully deleted
I20240502 21:26:08.366948 1420 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:08.368916 1157 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:08.378180 32107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:08.379927 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:08.413379 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:08.436103 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:08.445621 32242 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:08.511745 1177 consensus_queue.cc:237] T 3843ac68594f43079a1253a877c48065 P 21e73dc1a0eb486db9717fa320347e1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "21e73dc1a0eb486db9717fa320347e1c" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 36679 } }
I20240502 21:26:08.513007 2790 raft_consensus.cc:2932] T 3843ac68594f43079a1253a877c48065 P 21e73dc1a0eb486db9717fa320347e1c [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "21e73dc1a0eb486db9717fa320347e1c" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 36679 } } }
I20240502 21:26:08.517725 31978 catalog_manager.cc:5484] T 3843ac68594f43079a1253a877c48065 P 21e73dc1a0eb486db9717fa320347e1c reported cstate change: config changed from index 4 to 6, VOTER f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66) evicted. New cstate: current_term: 2 leader_uuid: "21e73dc1a0eb486db9717fa320347e1c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "21e73dc1a0eb486db9717fa320347e1c" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 36679 } } }
I20240502 21:26:08.521800 32242 tablet_service.cc:1514] Processing DeleteTablet for tablet 3843ac68594f43079a1253a877c48065 with delete_type TABLET_DATA_TOMBSTONED (TS f6f8b5e2067c4c23b0c89a8c471626f5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46274
I20240502 21:26:08.522562 3208 tablet_replica.cc:330] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5: stopping tablet replica
I20240502 21:26:08.522768 3208 raft_consensus.cc:2229] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:08.522987 3208 raft_consensus.cc:2258] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:08.523702 3208 ts_tablet_manager.cc:1875] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:08.528616 3208 ts_tablet_manager.cc:1888] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 21:26:08.528730 3208 log.cc:1196] T 3843ac68594f43079a1253a877c48065 P f6f8b5e2067c4c23b0c89a8c471626f5: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/3843ac68594f43079a1253a877c48065
I20240502 21:26:08.529558 31964 catalog_manager.cc:4830] TS f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66:39503): tablet 3843ac68594f43079a1253a877c48065 (table rebalance_test_table_0 [id=7a5ba90921d043e6a70466c331f7ef93]) successfully deleted
I20240502 21:26:08.632838 1420 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:08.645159 1157 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:08.647254 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:08.648236 32107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:08.664973 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:08.668063 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:08.668789 32242 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:08.757494 1177 consensus_queue.cc:237] T a80b4f90d4b04e1886e494cf23a6284b P 21e73dc1a0eb486db9717fa320347e1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "21e73dc1a0eb486db9717fa320347e1c" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 36679 } }
I20240502 21:26:08.758668 2790 raft_consensus.cc:2932] T a80b4f90d4b04e1886e494cf23a6284b P 21e73dc1a0eb486db9717fa320347e1c [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "21e73dc1a0eb486db9717fa320347e1c" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 36679 } } }
I20240502 21:26:08.762554 31971 catalog_manager.cc:5484] T a80b4f90d4b04e1886e494cf23a6284b P 21e73dc1a0eb486db9717fa320347e1c reported cstate change: config changed from index 4 to 6, VOTER f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66) evicted. New cstate: current_term: 2 leader_uuid: "21e73dc1a0eb486db9717fa320347e1c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "21e73dc1a0eb486db9717fa320347e1c" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 36679 } } }
I20240502 21:26:08.766345 32242 tablet_service.cc:1514] Processing DeleteTablet for tablet a80b4f90d4b04e1886e494cf23a6284b with delete_type TABLET_DATA_TOMBSTONED (TS f6f8b5e2067c4c23b0c89a8c471626f5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46274
I20240502 21:26:08.766822 3208 tablet_replica.cc:330] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5: stopping tablet replica
I20240502 21:26:08.767045 3208 raft_consensus.cc:2229] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:08.767278 3208 raft_consensus.cc:2258] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:08.767921 3208 ts_tablet_manager.cc:1875] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:08.772974 3208 ts_tablet_manager.cc:1888] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 21:26:08.773082 3208 log.cc:1196] T a80b4f90d4b04e1886e494cf23a6284b P f6f8b5e2067c4c23b0c89a8c471626f5: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/a80b4f90d4b04e1886e494cf23a6284b
I20240502 21:26:08.773803 31964 catalog_manager.cc:4830] TS f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66:39503): tablet a80b4f90d4b04e1886e494cf23a6284b (table rebalance_test_table_2 [id=bff79bbad6b54d32b0ee9781a39e2579]) successfully deleted
I20240502 21:26:08.864997 1157 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:08.873471 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:08.875154 1420 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:08.881620 32107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:08.897677 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:08.922087 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:08.923687 32242 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:09.012050 1440 consensus_queue.cc:237] T cd16f492f22c478a946d1da05f6a91b4 P 98ac0bd5300e42c38bee7aae9634c7c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "98ac0bd5300e42c38bee7aae9634c7c9" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 39493 } }
I20240502 21:26:09.013408 3294 raft_consensus.cc:2932] T cd16f492f22c478a946d1da05f6a91b4 P 98ac0bd5300e42c38bee7aae9634c7c9 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "98ac0bd5300e42c38bee7aae9634c7c9" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 39493 } } }
I20240502 21:26:09.018234 31975 catalog_manager.cc:5484] T cd16f492f22c478a946d1da05f6a91b4 P 98ac0bd5300e42c38bee7aae9634c7c9 reported cstate change: config changed from index 4 to 6, VOTER f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66) evicted. New cstate: current_term: 2 leader_uuid: "98ac0bd5300e42c38bee7aae9634c7c9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "98ac0bd5300e42c38bee7aae9634c7c9" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 39493 } } }
I20240502 21:26:09.021891 32242 tablet_service.cc:1514] Processing DeleteTablet for tablet cd16f492f22c478a946d1da05f6a91b4 with delete_type TABLET_DATA_TOMBSTONED (TS f6f8b5e2067c4c23b0c89a8c471626f5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46274
I20240502 21:26:09.022265 3208 tablet_replica.cc:330] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5: stopping tablet replica
I20240502 21:26:09.022498 3208 raft_consensus.cc:2229] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:09.022696 3208 raft_consensus.cc:2258] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:09.023345 3208 ts_tablet_manager.cc:1875] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:09.028059 3208 ts_tablet_manager.cc:1888] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 21:26:09.028173 3208 log.cc:1196] T cd16f492f22c478a946d1da05f6a91b4 P f6f8b5e2067c4c23b0c89a8c471626f5: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/cd16f492f22c478a946d1da05f6a91b4
I20240502 21:26:09.028903 31964 catalog_manager.cc:4830] TS f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66:39503): tablet cd16f492f22c478a946d1da05f6a91b4 (table rebalance_test_table_4 [id=0e0c0c30a67c4461bafc601a7c3a4676]) successfully deleted
I20240502 21:26:09.413565 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:09.427875 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:09.428120 1157 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:09.430121 32242 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:09.448927 32107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:09.449748 1420 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:09.450044 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:09.614699 1157 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:09.623028 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:09.630664 1420 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:09.645711 32107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:09.647081 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:09.652451 32242 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:09.671391 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:10.450172 1157 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:10.456058 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:10.467658 32107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:10.475061 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:10.487097 1420 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:10.487233 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:10.524022 32242 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:10.722599 1157 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:10.727330 1420 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:10.730634 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:10.731037 32107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:10.749567 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:10.755936 32242 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:10.768440 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:11.041584 1420 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:11.048288 1157 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:11.057245 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:11.060900 32107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:11.086958 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:11.099509 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:11.107228 32242 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
3efdd8a7bacc4111a711c88a6c613c1b | 127.29.82.126:44793 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.29.82.84:43193 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
21e73dc1a0eb486db9717fa320347e1c | 127.29.82.69:36679 | HEALTHY | <none> | 2 | 0
98ac0bd5300e42c38bee7aae9634c7c9 | 127.29.82.71:39493 | HEALTHY | <none> | 2 | 0
b399bf9ec0c544749c7eead22ddfedf6 | 127.29.82.67:45329 | HEALTHY | <none> | 2 | 0
cb8caceef23e475f8304b1ec1175e4b9 | 127.29.82.65:37947 | HEALTHY | <none> | 3 | 0
e705f0dda7904364a7453ee6294b3601 | 127.29.82.70:45931 | HEALTHY | <none> | 2 | 0
f58850eb82214ad39880e45777bf51b4 | 127.29.82.68:33951 | HEALTHY | <none> | 2 | 0
f6f8b5e2067c4c23b0c89a8c471626f5 | 127.29.82.66:39503 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.29.82.65 | experimental | 127.29.82.65:37947
local_ip_for_outbound_sockets | 127.29.82.66 | experimental | 127.29.82.66:39503
local_ip_for_outbound_sockets | 127.29.82.67 | experimental | 127.29.82.67:45329
local_ip_for_outbound_sockets | 127.29.82.68 | experimental | 127.29.82.68:33951
local_ip_for_outbound_sockets | 127.29.82.69 | experimental | 127.29.82.69:36679
local_ip_for_outbound_sockets | 127.29.82.70 | experimental | 127.29.82.70:45931
local_ip_for_outbound_sockets | 127.29.82.71 | experimental | 127.29.82.71:39493
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.82.65:37947
server_dump_info_path | /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.82.66:39503
server_dump_info_path | /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.82.67:45329
server_dump_info_path | /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.29.82.68:33951
server_dump_info_path | /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.82.69:36679
server_dump_info_path | /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.82.70:45931
server_dump_info_path | /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.29.82.71:39493
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.29.82.84:43193 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 21:26:11.156235 30025 log_verifier.cc:126] Checking tablet 3843ac68594f43079a1253a877c48065
I20240502 21:26:11.171825 30025 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 3843ac68594f43079a1253a877c48065
I20240502 21:26:11.171969 30025 log_verifier.cc:126] Checking tablet 43d4ba4bfe70416bb9c06a3bfe299835
I20240502 21:26:11.184621 30025 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 43d4ba4bfe70416bb9c06a3bfe299835
I20240502 21:26:11.184720 30025 log_verifier.cc:126] Checking tablet 4919235c52834df5b3924d93391f171a
I20240502 21:26:11.197170 30025 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 4919235c52834df5b3924d93391f171a
I20240502 21:26:11.197283 30025 log_verifier.cc:126] Checking tablet 4f1e70e3aa9940f3a6515bc61569565c
I20240502 21:26:11.208753 30025 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 4f1e70e3aa9940f3a6515bc61569565c
I20240502 21:26:11.208854 30025 log_verifier.cc:126] Checking tablet 60735476392d4c1492e9227c7f72e18a
I20240502 21:26:11.220520 30025 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 60735476392d4c1492e9227c7f72e18a
I20240502 21:26:11.220620 30025 log_verifier.cc:126] Checking tablet 699c3a3f730248bdacf122126883f2f4
I20240502 21:26:11.232636 30025 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 699c3a3f730248bdacf122126883f2f4
I20240502 21:26:11.232750 30025 log_verifier.cc:126] Checking tablet 7ee3c779759a49fd9db84c4f6b73e670
I20240502 21:26:11.244408 30025 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7ee3c779759a49fd9db84c4f6b73e670
I20240502 21:26:11.244534 30025 log_verifier.cc:126] Checking tablet 80471cfd4a944776baf05c4179f64a10
I20240502 21:26:11.256508 30025 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 80471cfd4a944776baf05c4179f64a10
I20240502 21:26:11.256613 30025 log_verifier.cc:126] Checking tablet 9812cd5d31154807a4f605eb615553f1
I20240502 21:26:11.264288 30025 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 9812cd5d31154807a4f605eb615553f1
I20240502 21:26:11.264393 30025 log_verifier.cc:126] Checking tablet 9873952de1224793b8b7f17ef92295b7
I20240502 21:26:11.276684 30025 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 9873952de1224793b8b7f17ef92295b7
I20240502 21:26:11.276794 30025 log_verifier.cc:126] Checking tablet a80b4f90d4b04e1886e494cf23a6284b
I20240502 21:26:11.289059 30025 log_verifier.cc:177] Verified matching terms for 6 ops in tablet a80b4f90d4b04e1886e494cf23a6284b
I20240502 21:26:11.289170 30025 log_verifier.cc:126] Checking tablet bc9b263fc4224e8eb8ca3c82a808bce8
I20240502 21:26:11.300927 30025 log_verifier.cc:177] Verified matching terms for 3 ops in tablet bc9b263fc4224e8eb8ca3c82a808bce8
I20240502 21:26:11.301036 30025 log_verifier.cc:126] Checking tablet cc8ba66c2b954876891bdded155d2b91
I20240502 21:26:11.312651 30025 log_verifier.cc:177] Verified matching terms for 3 ops in tablet cc8ba66c2b954876891bdded155d2b91
I20240502 21:26:11.312774 30025 log_verifier.cc:126] Checking tablet cd16f492f22c478a946d1da05f6a91b4
I20240502 21:26:11.325004 30025 log_verifier.cc:177] Verified matching terms for 6 ops in tablet cd16f492f22c478a946d1da05f6a91b4
I20240502 21:26:11.325126 30025 log_verifier.cc:126] Checking tablet d484c2aeb0904564997462a634b541d3
I20240502 21:26:11.337462 30025 log_verifier.cc:177] Verified matching terms for 6 ops in tablet d484c2aeb0904564997462a634b541d3
I20240502 21:26:11.337563 30025 log_verifier.cc:126] Checking tablet e60fc6b2f53444869f20be78f7c7cd6e
I20240502 21:26:11.348875 30025 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e60fc6b2f53444869f20be78f7c7cd6e
I20240502 21:26:11.352293 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.72:0
--local_ip_for_outbound_sockets=127.29.82.72
--webserver_interface=127.29.82.72
--webserver_port=0
--tserver_master_addrs=127.29.82.126:44793
--builtin_ntp_servers=127.29.82.84:43193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 21:26:11.690975 3547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:11.691269 3547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:11.691366 3547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:11.698985 3547 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:11.699100 3547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:11.699244 3547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.72
I20240502 21:26:11.708391 3547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:43193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.29.82.72
--webserver_port=0
--tserver_master_addrs=127.29.82.126:44793
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.72
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:11.709721 3547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:11.711478 3547 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:11.725225 3555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:11.726188 3553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:11.727074 3552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:11.728375 3547 server_base.cc:1034] running on GCE node
I20240502 21:26:11.728917 3547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:11.729777 3547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:11.730983 3547 hybrid_clock.cc:648] HybridClock initialized: now 1714685171730937 us; error 33 us; skew 500 ppm
I20240502 21:26:11.731299 3547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:11.733456 3547 webserver.cc:460] Webserver started at http://127.29.82.72:32769/ using document root <none> and password file <none>
I20240502 21:26:11.734161 3547 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:11.734270 3547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:11.734588 3547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:11.737277 3547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "e7352d334c9a4bc69910e53e94348acf"
format_stamp: "Formatted at 2024-05-02 21:26:11 on dist-test-slave-xzsm"
I20240502 21:26:11.738090 3547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "e7352d334c9a4bc69910e53e94348acf"
format_stamp: "Formatted at 2024-05-02 21:26:11 on dist-test-slave-xzsm"
I20240502 21:26:11.743090 3547 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20240502 21:26:11.746433 3561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:11.747676 3547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:11.747818 3547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "e7352d334c9a4bc69910e53e94348acf"
format_stamp: "Formatted at 2024-05-02 21:26:11 on dist-test-slave-xzsm"
I20240502 21:26:11.747993 3547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:11.772897 3547 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:11.773644 3547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:11.773914 3547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:11.774796 3547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:11.776165 3547 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:11.776252 3547 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:11.776335 3547 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:11.776391 3547 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:11.808719 3547 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.72:39907
I20240502 21:26:11.808753 3674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.72:39907 every 8 connection(s)
I20240502 21:26:11.810050 3547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 21:26:11.813951 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 3547
I20240502 21:26:11.814205 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 21:26:11.825785 3675 heartbeater.cc:347] Connected to a master server at 127.29.82.126:44793
I20240502 21:26:11.826117 3675 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:11.827030 3675 heartbeater.cc:511] Master 127.29.82.126:44793 requested a full tablet report, sending...
I20240502 21:26:11.828575 31974 ts_manager.cc:194] Registered new tserver with Master: e7352d334c9a4bc69910e53e94348acf (127.29.82.72:39907)
I20240502 21:26:11.829455 31974 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.72:43563
I20240502 21:26:12.306805 1157 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:12.307268 1420 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:12.312961 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:12.333959 32107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:12.338675 3609 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:12.339521 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:12.377609 32242 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:12.386520 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:12.573088 1157 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:12.576282 1420 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:12.590248 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:12.592139 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:12.602066 3609 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:12.604074 32107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:12.621300 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:12.637243 32242 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:12.735361 32263 consensus_queue.cc:237] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "e7352d334c9a4bc69910e53e94348acf" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 39907 } }
W20240502 21:26:12.743933 32196 consensus_peers.cc:485] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5 -> Peer e7352d334c9a4bc69910e53e94348acf (127.29.82.72:39907): Couldn't send request to peer e7352d334c9a4bc69910e53e94348acf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e60fc6b2f53444869f20be78f7c7cd6e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:12.832444 3675 heartbeater.cc:503] Master 127.29.82.126:44793 was elected leader, sending a full tablet report...
I20240502 21:26:13.298775 3782 ts_tablet_manager.cc:902] T e60fc6b2f53444869f20be78f7c7cd6e P e7352d334c9a4bc69910e53e94348acf: Initiating tablet copy from peer f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66:39503)
I20240502 21:26:13.300465 3782 tablet_copy_client.cc:308] T e60fc6b2f53444869f20be78f7c7cd6e P e7352d334c9a4bc69910e53e94348acf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.66:39503
I20240502 21:26:13.306852 32283 tablet_copy_service.cc:140] P f6f8b5e2067c4c23b0c89a8c471626f5: Received BeginTabletCopySession request for tablet e60fc6b2f53444869f20be78f7c7cd6e from peer e7352d334c9a4bc69910e53e94348acf ({username='slave'} at 127.29.82.72:59845)
I20240502 21:26:13.307049 32283 tablet_copy_service.cc:161] P f6f8b5e2067c4c23b0c89a8c471626f5: Beginning new tablet copy session on tablet e60fc6b2f53444869f20be78f7c7cd6e from peer e7352d334c9a4bc69910e53e94348acf at {username='slave'} at 127.29.82.72:59845: session id = e7352d334c9a4bc69910e53e94348acf-e60fc6b2f53444869f20be78f7c7cd6e
I20240502 21:26:13.309056 32283 tablet_copy_source_session.cc:205] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:13.313318 3782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e60fc6b2f53444869f20be78f7c7cd6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:13.322872 3782 tablet_copy_client.cc:783] T e60fc6b2f53444869f20be78f7c7cd6e P e7352d334c9a4bc69910e53e94348acf: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:13.323202 3782 tablet_copy_client.cc:647] T e60fc6b2f53444869f20be78f7c7cd6e P e7352d334c9a4bc69910e53e94348acf: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:13.325307 3782 tablet_copy_client.cc:523] T e60fc6b2f53444869f20be78f7c7cd6e P e7352d334c9a4bc69910e53e94348acf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:13.328282 3782 tablet_bootstrap.cc:492] T e60fc6b2f53444869f20be78f7c7cd6e P e7352d334c9a4bc69910e53e94348acf: Bootstrap starting.
I20240502 21:26:13.337085 3782 log.cc:824] T e60fc6b2f53444869f20be78f7c7cd6e P e7352d334c9a4bc69910e53e94348acf: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:13.373867 3782 tablet_bootstrap.cc:492] T e60fc6b2f53444869f20be78f7c7cd6e P e7352d334c9a4bc69910e53e94348acf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 21:26:13.374428 3782 tablet_bootstrap.cc:492] T e60fc6b2f53444869f20be78f7c7cd6e P e7352d334c9a4bc69910e53e94348acf: Bootstrap complete.
I20240502 21:26:13.374845 3782 ts_tablet_manager.cc:1367] T e60fc6b2f53444869f20be78f7c7cd6e P e7352d334c9a4bc69910e53e94348acf: Time spent bootstrapping tablet: real 0.047s user 0.044s sys 0.000s
I20240502 21:26:13.379137 3782 raft_consensus.cc:348] T e60fc6b2f53444869f20be78f7c7cd6e P e7352d334c9a4bc69910e53e94348acf [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } }
I20240502 21:26:13.379979 3782 raft_consensus.cc:729] T e60fc6b2f53444869f20be78f7c7cd6e P e7352d334c9a4bc69910e53e94348acf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7352d334c9a4bc69910e53e94348acf, State: Initialized, Role: FOLLOWER
I20240502 21:26:13.380380 3782 consensus_queue.cc:260] T e60fc6b2f53444869f20be78f7c7cd6e P e7352d334c9a4bc69910e53e94348acf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "e7352d334c9a4bc69910e53e94348acf" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 39907 } }
I20240502 21:26:13.381989 3782 ts_tablet_manager.cc:1398] T e60fc6b2f53444869f20be78f7c7cd6e P e7352d334c9a4bc69910e53e94348acf: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20240502 21:26:13.383009 32283 tablet_copy_service.cc:342] P f6f8b5e2067c4c23b0c89a8c471626f5: Request end of tablet copy session e7352d334c9a4bc69910e53e94348acf-e60fc6b2f53444869f20be78f7c7cd6e received from {username='slave'} at 127.29.82.72:59845
I20240502 21:26:13.383131 32283 tablet_copy_service.cc:433] P f6f8b5e2067c4c23b0c89a8c471626f5: ending tablet copy session e7352d334c9a4bc69910e53e94348acf-e60fc6b2f53444869f20be78f7c7cd6e on tablet e60fc6b2f53444869f20be78f7c7cd6e with peer e7352d334c9a4bc69910e53e94348acf
I20240502 21:26:13.880292 3629 raft_consensus.cc:1203] T e60fc6b2f53444869f20be78f7c7cd6e P e7352d334c9a4bc69910e53e94348acf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:26:13.882099 3792 raft_consensus.cc:2932] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e7352d334c9a4bc69910e53e94348acf (127.29.82.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "e7352d334c9a4bc69910e53e94348acf" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 39907 } } }
I20240502 21:26:13.882844 3629 raft_consensus.cc:2932] T e60fc6b2f53444869f20be78f7c7cd6e P e7352d334c9a4bc69910e53e94348acf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e7352d334c9a4bc69910e53e94348acf (127.29.82.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "e7352d334c9a4bc69910e53e94348acf" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 39907 } } }
I20240502 21:26:13.885507 3792 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:53170 (request call id 0) took 1150 ms. Trace:
I20240502 21:26:13.885735 3792 rpcz_store.cc:276] 0502 21:26:12.734820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:26:12.734890 (+ 70us) service_pool.cc:225] Handling call
0502 21:26:12.736194 (+ 1304us) log.cc:842] Serialized 257 byte log entry
0502 21:26:13.885455 (+1149261us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":83,"raft.queue_time_us":428,"raft.run_cpu_time_us":315,"raft.run_wall_time_us":315,"thread_start_us":519,"threads_started":2,"wal-append.queue_time_us":302,"wal-append.run_cpu_time_us":692,"wal-append.run_wall_time_us":1000717}
I20240502 21:26:13.888183 31980 catalog_manager.cc:5484] T e60fc6b2f53444869f20be78f7c7cd6e P e7352d334c9a4bc69910e53e94348acf reported cstate change: config changed from index -1 to 4, VOTER e7352d334c9a4bc69910e53e94348acf (127.29.82.72) added. New cstate: current_term: 1 leader_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "e7352d334c9a4bc69910e53e94348acf" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 39907 } } }
I20240502 21:26:13.919626 32126 consensus_queue.cc:237] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "e7352d334c9a4bc69910e53e94348acf" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 39907 } }
W20240502 21:26:13.928395 32061 consensus_peers.cc:485] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9 -> Peer e7352d334c9a4bc69910e53e94348acf (127.29.82.72:39907): Couldn't send request to peer e7352d334c9a4bc69910e53e94348acf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60735476392d4c1492e9227c7f72e18a. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:14.392475 3821 ts_tablet_manager.cc:902] T 60735476392d4c1492e9227c7f72e18a P e7352d334c9a4bc69910e53e94348acf: Initiating tablet copy from peer cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65:37947)
I20240502 21:26:14.393782 3821 tablet_copy_client.cc:308] T 60735476392d4c1492e9227c7f72e18a P e7352d334c9a4bc69910e53e94348acf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.65:37947
I20240502 21:26:14.401890 32147 tablet_copy_service.cc:140] P cb8caceef23e475f8304b1ec1175e4b9: Received BeginTabletCopySession request for tablet 60735476392d4c1492e9227c7f72e18a from peer e7352d334c9a4bc69910e53e94348acf ({username='slave'} at 127.29.82.72:56121)
I20240502 21:26:14.402066 32147 tablet_copy_service.cc:161] P cb8caceef23e475f8304b1ec1175e4b9: Beginning new tablet copy session on tablet 60735476392d4c1492e9227c7f72e18a from peer e7352d334c9a4bc69910e53e94348acf at {username='slave'} at 127.29.82.72:56121: session id = e7352d334c9a4bc69910e53e94348acf-60735476392d4c1492e9227c7f72e18a
I20240502 21:26:14.404109 32147 tablet_copy_source_session.cc:205] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:14.405407 3821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60735476392d4c1492e9227c7f72e18a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:14.409837 3821 tablet_copy_client.cc:783] T 60735476392d4c1492e9227c7f72e18a P e7352d334c9a4bc69910e53e94348acf: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:14.410058 3821 tablet_copy_client.cc:647] T 60735476392d4c1492e9227c7f72e18a P e7352d334c9a4bc69910e53e94348acf: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:14.411847 3821 tablet_copy_client.cc:523] T 60735476392d4c1492e9227c7f72e18a P e7352d334c9a4bc69910e53e94348acf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:14.414287 3821 tablet_bootstrap.cc:492] T 60735476392d4c1492e9227c7f72e18a P e7352d334c9a4bc69910e53e94348acf: Bootstrap starting.
I20240502 21:26:14.447001 3821 tablet_bootstrap.cc:492] T 60735476392d4c1492e9227c7f72e18a P e7352d334c9a4bc69910e53e94348acf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 21:26:14.447386 3821 tablet_bootstrap.cc:492] T 60735476392d4c1492e9227c7f72e18a P e7352d334c9a4bc69910e53e94348acf: Bootstrap complete.
I20240502 21:26:14.447659 3821 ts_tablet_manager.cc:1367] T 60735476392d4c1492e9227c7f72e18a P e7352d334c9a4bc69910e53e94348acf: Time spent bootstrapping tablet: real 0.033s user 0.035s sys 0.000s
I20240502 21:26:14.448488 3821 raft_consensus.cc:348] T 60735476392d4c1492e9227c7f72e18a P e7352d334c9a4bc69910e53e94348acf [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } }
I20240502 21:26:14.448841 3821 raft_consensus.cc:729] T 60735476392d4c1492e9227c7f72e18a P e7352d334c9a4bc69910e53e94348acf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7352d334c9a4bc69910e53e94348acf, State: Initialized, Role: FOLLOWER
I20240502 21:26:14.449000 3821 consensus_queue.cc:260] T 60735476392d4c1492e9227c7f72e18a P e7352d334c9a4bc69910e53e94348acf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "e7352d334c9a4bc69910e53e94348acf" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 39907 } }
I20240502 21:26:14.449681 3821 ts_tablet_manager.cc:1398] T 60735476392d4c1492e9227c7f72e18a P e7352d334c9a4bc69910e53e94348acf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:14.450647 32147 tablet_copy_service.cc:342] P cb8caceef23e475f8304b1ec1175e4b9: Request end of tablet copy session e7352d334c9a4bc69910e53e94348acf-60735476392d4c1492e9227c7f72e18a received from {username='slave'} at 127.29.82.72:56121
I20240502 21:26:14.450803 32147 tablet_copy_service.cc:433] P cb8caceef23e475f8304b1ec1175e4b9: ending tablet copy session e7352d334c9a4bc69910e53e94348acf-60735476392d4c1492e9227c7f72e18a on tablet 60735476392d4c1492e9227c7f72e18a with peer e7352d334c9a4bc69910e53e94348acf
I20240502 21:26:14.993543 3629 raft_consensus.cc:1203] T 60735476392d4c1492e9227c7f72e18a P e7352d334c9a4bc69910e53e94348acf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:26:14.995158 3831 raft_consensus.cc:2932] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e7352d334c9a4bc69910e53e94348acf (127.29.82.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "e7352d334c9a4bc69910e53e94348acf" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 39907 } } }
I20240502 21:26:14.995875 3629 raft_consensus.cc:2932] T 60735476392d4c1492e9227c7f72e18a P e7352d334c9a4bc69910e53e94348acf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e7352d334c9a4bc69910e53e94348acf (127.29.82.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "e7352d334c9a4bc69910e53e94348acf" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 39907 } } }
I20240502 21:26:14.998970 3831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:33942 (request call id 0) took 1080 ms. Trace:
I20240502 21:26:14.999176 3831 rpcz_store.cc:276] 0502 21:26:13.918951 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:26:13.919025 (+ 74us) service_pool.cc:225] Handling call
0502 21:26:13.920429 (+ 1404us) log.cc:842] Serialized 257 byte log entry
0502 21:26:14.998912 (+1078483us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":76,"raft.queue_time_us":380,"raft.run_cpu_time_us":299,"raft.run_wall_time_us":299,"thread_start_us":483,"threads_started":2,"wal-append.queue_time_us":330,"wal-append.run_cpu_time_us":638,"wal-append.run_wall_time_us":1000705}
I20240502 21:26:15.001161 31971 catalog_manager.cc:5484] T 60735476392d4c1492e9227c7f72e18a P e7352d334c9a4bc69910e53e94348acf reported cstate change: config changed from index -1 to 4, VOTER e7352d334c9a4bc69910e53e94348acf (127.29.82.72) added. New cstate: current_term: 1 leader_uuid: "cb8caceef23e475f8304b1ec1175e4b9" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "e7352d334c9a4bc69910e53e94348acf" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 39907 } } }
I20240502 21:26:15.125648 1420 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:15.128509 32107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:15.140004 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:15.151602 1157 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:15.190542 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:15.191344 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:15.191797 3609 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:15.234830 32242 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:15.273897 32126 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "60735476392d4c1492e9227c7f72e18a"
dest_uuid: "cb8caceef23e475f8304b1ec1175e4b9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33976
I20240502 21:26:15.274123 32126 raft_consensus.cc:595] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:26:15.365073 1157 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:15.369442 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:15.372359 1420 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:15.383105 32107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:15.390442 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:15.403354 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:15.443802 3609 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:15.453341 32242 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:15.456679 3831 raft_consensus.cc:979] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9: : Instructing follower e7352d334c9a4bc69910e53e94348acf to start an election
I20240502 21:26:15.456857 3831 raft_consensus.cc:1067] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Signalling peer e7352d334c9a4bc69910e53e94348acf to start an election
I20240502 21:26:15.457916 3629 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "60735476392d4c1492e9227c7f72e18a"
dest_uuid: "e7352d334c9a4bc69910e53e94348acf"
from {username='slave'} at 127.29.82.65:49905
I20240502 21:26:15.458191 3629 raft_consensus.cc:482] T 60735476392d4c1492e9227c7f72e18a P e7352d334c9a4bc69910e53e94348acf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:26:15.458382 3629 raft_consensus.cc:3037] T 60735476392d4c1492e9227c7f72e18a P e7352d334c9a4bc69910e53e94348acf [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:15.461424 3629 raft_consensus.cc:504] T 60735476392d4c1492e9227c7f72e18a P e7352d334c9a4bc69910e53e94348acf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "e7352d334c9a4bc69910e53e94348acf" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 39907 } }
I20240502 21:26:15.463362 32126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60735476392d4c1492e9227c7f72e18a" candidate_uuid: "e7352d334c9a4bc69910e53e94348acf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "cb8caceef23e475f8304b1ec1175e4b9"
I20240502 21:26:15.463626 32126 raft_consensus.cc:3032] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:26:15.463704 32126 raft_consensus.cc:729] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cb8caceef23e475f8304b1ec1175e4b9, State: Running, Role: LEADER
I20240502 21:26:15.463958 32126 consensus_queue.cc:260] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "e7352d334c9a4bc69910e53e94348acf" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 39907 } }
I20240502 21:26:15.464148 32126 raft_consensus.cc:3037] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:15.466856 32126 raft_consensus.cc:2454] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7352d334c9a4bc69910e53e94348acf in term 2.
I20240502 21:26:15.467567 3565 leader_election.cc:302] T 60735476392d4c1492e9227c7f72e18a P e7352d334c9a4bc69910e53e94348acf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: cb8caceef23e475f8304b1ec1175e4b9, e7352d334c9a4bc69910e53e94348acf; no voters:
I20240502 21:26:15.468001 3830 raft_consensus.cc:2781] T 60735476392d4c1492e9227c7f72e18a P e7352d334c9a4bc69910e53e94348acf [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:26:15.468641 3629 leader_election.cc:288] T 60735476392d4c1492e9227c7f72e18a P e7352d334c9a4bc69910e53e94348acf [CANDIDATE]: Term 2 election: Requested vote from peers cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65:37947)
I20240502 21:26:15.483115 3830 raft_consensus.cc:686] T 60735476392d4c1492e9227c7f72e18a P e7352d334c9a4bc69910e53e94348acf [term 2 LEADER]: Becoming Leader. State: Replica: e7352d334c9a4bc69910e53e94348acf, State: Running, Role: LEADER
I20240502 21:26:15.483898 3830 consensus_queue.cc:237] T 60735476392d4c1492e9227c7f72e18a P e7352d334c9a4bc69910e53e94348acf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "e7352d334c9a4bc69910e53e94348acf" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 39907 } }
I20240502 21:26:15.488133 31978 catalog_manager.cc:5484] T 60735476392d4c1492e9227c7f72e18a P e7352d334c9a4bc69910e53e94348acf reported cstate change: term changed from 1 to 2, leader changed from cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65) to e7352d334c9a4bc69910e53e94348acf (127.29.82.72). New cstate: current_term: 2 leader_uuid: "e7352d334c9a4bc69910e53e94348acf" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 } } peers { permanent_uuid: "e7352d334c9a4bc69910e53e94348acf" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 39907 } } }
I20240502 21:26:15.635886 1157 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:15.639516 1420 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:15.656211 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:15.656947 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:15.658941 3609 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:15.677628 32107 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:15.711426 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:15.746069 32242 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:15.789040 32263 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e60fc6b2f53444869f20be78f7c7cd6e"
dest_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58874
I20240502 21:26:15.789273 32263 raft_consensus.cc:595] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:26:15.886986 1157 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:15.888332 1420 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:15.907763 3609 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:15.908990 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:15.917630 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:15.918061 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:15.923130 32242 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:15.952514 32107 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:16.090122 32126 raft_consensus.cc:1261] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9 [term 2 FOLLOWER]: Refusing update from remote peer e7352d334c9a4bc69910e53e94348acf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:26:16.090888 4008 consensus_queue.cc:1035] T 60735476392d4c1492e9227c7f72e18a P e7352d334c9a4bc69910e53e94348acf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb8caceef23e475f8304b1ec1175e4b9" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 37947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:16.125137 4015 raft_consensus.cc:979] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5: : Instructing follower e7352d334c9a4bc69910e53e94348acf to start an election
I20240502 21:26:16.125367 4015 raft_consensus.cc:1067] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Signalling peer e7352d334c9a4bc69910e53e94348acf to start an election
I20240502 21:26:16.126158 3629 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e60fc6b2f53444869f20be78f7c7cd6e"
dest_uuid: "e7352d334c9a4bc69910e53e94348acf"
from {username='slave'} at 127.29.82.66:45685
I20240502 21:26:16.126435 3629 raft_consensus.cc:482] T e60fc6b2f53444869f20be78f7c7cd6e P e7352d334c9a4bc69910e53e94348acf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:26:16.126565 3629 raft_consensus.cc:3037] T e60fc6b2f53444869f20be78f7c7cd6e P e7352d334c9a4bc69910e53e94348acf [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:16.128522 3629 raft_consensus.cc:504] T e60fc6b2f53444869f20be78f7c7cd6e P e7352d334c9a4bc69910e53e94348acf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "e7352d334c9a4bc69910e53e94348acf" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 39907 } }
I20240502 21:26:16.129128 3629 leader_election.cc:288] T e60fc6b2f53444869f20be78f7c7cd6e P e7352d334c9a4bc69910e53e94348acf [CANDIDATE]: Term 2 election: Requested vote from peers f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66:39503)
I20240502 21:26:16.129634 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e60fc6b2f53444869f20be78f7c7cd6e" candidate_uuid: "e7352d334c9a4bc69910e53e94348acf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5"
I20240502 21:26:16.129832 32263 raft_consensus.cc:3032] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:26:16.129904 32263 raft_consensus.cc:729] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6f8b5e2067c4c23b0c89a8c471626f5, State: Running, Role: LEADER
I20240502 21:26:16.130149 32263 consensus_queue.cc:260] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "e7352d334c9a4bc69910e53e94348acf" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 39907 } }
I20240502 21:26:16.130394 32263 raft_consensus.cc:3037] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:16.132218 32263 raft_consensus.cc:2454] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7352d334c9a4bc69910e53e94348acf in term 2.
I20240502 21:26:16.132658 3562 leader_election.cc:302] T e60fc6b2f53444869f20be78f7c7cd6e P e7352d334c9a4bc69910e53e94348acf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: e7352d334c9a4bc69910e53e94348acf, f6f8b5e2067c4c23b0c89a8c471626f5; no voters:
I20240502 21:26:16.132944 4011 raft_consensus.cc:2781] T e60fc6b2f53444869f20be78f7c7cd6e P e7352d334c9a4bc69910e53e94348acf [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:26:16.133069 4011 raft_consensus.cc:686] T e60fc6b2f53444869f20be78f7c7cd6e P e7352d334c9a4bc69910e53e94348acf [term 2 LEADER]: Becoming Leader. State: Replica: e7352d334c9a4bc69910e53e94348acf, State: Running, Role: LEADER
I20240502 21:26:16.133307 4011 consensus_queue.cc:237] T e60fc6b2f53444869f20be78f7c7cd6e P e7352d334c9a4bc69910e53e94348acf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "e7352d334c9a4bc69910e53e94348acf" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 39907 } }
I20240502 21:26:16.135561 31971 catalog_manager.cc:5484] T e60fc6b2f53444869f20be78f7c7cd6e P e7352d334c9a4bc69910e53e94348acf reported cstate change: term changed from 1 to 2, leader changed from f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66) to e7352d334c9a4bc69910e53e94348acf (127.29.82.72). New cstate: current_term: 2 leader_uuid: "e7352d334c9a4bc69910e53e94348acf" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 } } peers { permanent_uuid: "e7352d334c9a4bc69910e53e94348acf" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 39907 } } }
I20240502 21:26:16.326396 1420 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:16.329219 1157 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:16.352821 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:16.362241 3609 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:16.364054 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:16.377125 32107 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:16.378662 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:16.396541 32242 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:16.486449 3629 consensus_queue.cc:237] T 60735476392d4c1492e9227c7f72e18a P e7352d334c9a4bc69910e53e94348acf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e7352d334c9a4bc69910e53e94348acf" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 39907 } }
I20240502 21:26:16.487293 4011 raft_consensus.cc:2932] T 60735476392d4c1492e9227c7f72e18a P e7352d334c9a4bc69910e53e94348acf [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e7352d334c9a4bc69910e53e94348acf" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 39907 } } }
I20240502 21:26:16.491786 31975 catalog_manager.cc:5484] T 60735476392d4c1492e9227c7f72e18a P e7352d334c9a4bc69910e53e94348acf reported cstate change: config changed from index 4 to 6, VOTER cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65) evicted. New cstate: current_term: 2 leader_uuid: "e7352d334c9a4bc69910e53e94348acf" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e7352d334c9a4bc69910e53e94348acf" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 39907 } } }
I20240502 21:26:16.495792 32107 tablet_service.cc:1514] Processing DeleteTablet for tablet 60735476392d4c1492e9227c7f72e18a with delete_type TABLET_DATA_TOMBSTONED (TS cb8caceef23e475f8304b1ec1175e4b9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35662
I20240502 21:26:16.496595 4081 tablet_replica.cc:330] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9: stopping tablet replica
I20240502 21:26:16.496805 4081 raft_consensus.cc:2229] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:16.497011 4081 raft_consensus.cc:2258] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:16.499011 4081 ts_tablet_manager.cc:1875] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:16.504169 4081 ts_tablet_manager.cc:1888] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 21:26:16.504269 4081 log.cc:1196] T 60735476392d4c1492e9227c7f72e18a P cb8caceef23e475f8304b1ec1175e4b9: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/60735476392d4c1492e9227c7f72e18a
I20240502 21:26:16.505023 31967 catalog_manager.cc:4830] TS cb8caceef23e475f8304b1ec1175e4b9 (127.29.82.65:37947): tablet 60735476392d4c1492e9227c7f72e18a (table rebalance_test_table_4 [id=0e0c0c30a67c4461bafc601a7c3a4676]) successfully deleted
I20240502 21:26:16.553442 32263 raft_consensus.cc:1261] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5 [term 2 FOLLOWER]: Refusing update from remote peer e7352d334c9a4bc69910e53e94348acf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:26:16.554167 4011 consensus_queue.cc:1035] T e60fc6b2f53444869f20be78f7c7cd6e P e7352d334c9a4bc69910e53e94348acf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6f8b5e2067c4c23b0c89a8c471626f5" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:16.613191 1420 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:16.616776 1157 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:16.619307 3609 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:16.620687 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:16.637100 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:16.638449 32107 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:16.666077 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:16.682898 32242 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:16.752993 3629 consensus_queue.cc:237] T e60fc6b2f53444869f20be78f7c7cd6e P e7352d334c9a4bc69910e53e94348acf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e7352d334c9a4bc69910e53e94348acf" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 39907 } }
I20240502 21:26:16.753896 4008 raft_consensus.cc:2932] T e60fc6b2f53444869f20be78f7c7cd6e P e7352d334c9a4bc69910e53e94348acf [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e7352d334c9a4bc69910e53e94348acf" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 39907 } } }
I20240502 21:26:16.758237 31978 catalog_manager.cc:5484] T e60fc6b2f53444869f20be78f7c7cd6e P e7352d334c9a4bc69910e53e94348acf reported cstate change: config changed from index 4 to 6, VOTER f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66) evicted. New cstate: current_term: 2 leader_uuid: "e7352d334c9a4bc69910e53e94348acf" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e7352d334c9a4bc69910e53e94348acf" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 39907 } } }
I20240502 21:26:16.761482 32242 tablet_service.cc:1514] Processing DeleteTablet for tablet e60fc6b2f53444869f20be78f7c7cd6e with delete_type TABLET_DATA_TOMBSTONED (TS f6f8b5e2067c4c23b0c89a8c471626f5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46274
I20240502 21:26:16.762085 4133 tablet_replica.cc:330] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5: stopping tablet replica
I20240502 21:26:16.762265 4133 raft_consensus.cc:2229] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:16.762449 4133 raft_consensus.cc:2258] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:16.763985 4133 ts_tablet_manager.cc:1875] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:16.768463 4133 ts_tablet_manager.cc:1888] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 21:26:16.768579 4133 log.cc:1196] T e60fc6b2f53444869f20be78f7c7cd6e P f6f8b5e2067c4c23b0c89a8c471626f5: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/e60fc6b2f53444869f20be78f7c7cd6e
I20240502 21:26:16.769289 31964 catalog_manager.cc:4830] TS f6f8b5e2067c4c23b0c89a8c471626f5 (127.29.82.66:39503): tablet e60fc6b2f53444869f20be78f7c7cd6e (table rebalance_test_table_3 [id=74a97a8ddca948248cc5d22c9b96616c]) successfully deleted
I20240502 21:26:17.097111 1157 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:17.101373 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:17.111505 1420 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:17.127085 32107 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:17.160113 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:17.178530 3609 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:17.233567 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:17.240497 32242 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:17.458194 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:17.478875 1420 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:17.478888 1157 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:17.481277 32107 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:17.489600 3609 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:17.508749 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:17.509495 32242 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:17.523525 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:17.847707 1420 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:17.849139 1157 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:17.857455 32107 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:17.865347 890 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:17.866353 3609 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:17.874781 1288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:17.883859 1024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:17.892976 32242 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
3efdd8a7bacc4111a711c88a6c613c1b | 127.29.82.126:44793 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.29.82.84:43193 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
21e73dc1a0eb486db9717fa320347e1c | 127.29.82.69:36679 | HEALTHY | <none> | 2 | 0
98ac0bd5300e42c38bee7aae9634c7c9 | 127.29.82.71:39493 | HEALTHY | <none> | 2 | 0
b399bf9ec0c544749c7eead22ddfedf6 | 127.29.82.67:45329 | HEALTHY | <none> | 2 | 0
cb8caceef23e475f8304b1ec1175e4b9 | 127.29.82.65:37947 | HEALTHY | <none> | 2 | 0
e705f0dda7904364a7453ee6294b3601 | 127.29.82.70:45931 | HEALTHY | <none> | 2 | 0
e7352d334c9a4bc69910e53e94348acf | 127.29.82.72:39907 | HEALTHY | <none> | 2 | 0
f58850eb82214ad39880e45777bf51b4 | 127.29.82.68:33951 | HEALTHY | <none> | 2 | 0
f6f8b5e2067c4c23b0c89a8c471626f5 | 127.29.82.66:39503 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.29.82.65 | experimental | 127.29.82.65:37947
local_ip_for_outbound_sockets | 127.29.82.66 | experimental | 127.29.82.66:39503
local_ip_for_outbound_sockets | 127.29.82.67 | experimental | 127.29.82.67:45329
local_ip_for_outbound_sockets | 127.29.82.68 | experimental | 127.29.82.68:33951
local_ip_for_outbound_sockets | 127.29.82.69 | experimental | 127.29.82.69:36679
local_ip_for_outbound_sockets | 127.29.82.70 | experimental | 127.29.82.70:45931
local_ip_for_outbound_sockets | 127.29.82.71 | experimental | 127.29.82.71:39493
local_ip_for_outbound_sockets | 127.29.82.72 | experimental | 127.29.82.72:39907
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.82.65:37947
server_dump_info_path | /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.82.66:39503
server_dump_info_path | /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.82.67:45329
server_dump_info_path | /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.29.82.68:33951
server_dump_info_path | /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.82.69:36679
server_dump_info_path | /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.82.70:45931
server_dump_info_path | /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.29.82.71:39493
server_dump_info_path | /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.29.82.72:39907
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.29.82.84:43193 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 21:26:17.980206 30025 log_verifier.cc:126] Checking tablet 3843ac68594f43079a1253a877c48065
I20240502 21:26:17.990823 30025 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 3843ac68594f43079a1253a877c48065
I20240502 21:26:17.990917 30025 log_verifier.cc:126] Checking tablet 43d4ba4bfe70416bb9c06a3bfe299835
I20240502 21:26:18.000669 30025 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 43d4ba4bfe70416bb9c06a3bfe299835
I20240502 21:26:18.000742 30025 log_verifier.cc:126] Checking tablet 4919235c52834df5b3924d93391f171a
I20240502 21:26:18.010300 30025 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 4919235c52834df5b3924d93391f171a
I20240502 21:26:18.010373 30025 log_verifier.cc:126] Checking tablet 4f1e70e3aa9940f3a6515bc61569565c
I20240502 21:26:18.022661 30025 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 4f1e70e3aa9940f3a6515bc61569565c
I20240502 21:26:18.022753 30025 log_verifier.cc:126] Checking tablet 60735476392d4c1492e9227c7f72e18a
I20240502 21:26:18.040526 30025 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 60735476392d4c1492e9227c7f72e18a
I20240502 21:26:18.040616 30025 log_verifier.cc:126] Checking tablet 699c3a3f730248bdacf122126883f2f4
I20240502 21:26:18.051105 30025 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 699c3a3f730248bdacf122126883f2f4
I20240502 21:26:18.051191 30025 log_verifier.cc:126] Checking tablet 7ee3c779759a49fd9db84c4f6b73e670
I20240502 21:26:18.060623 30025 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7ee3c779759a49fd9db84c4f6b73e670
I20240502 21:26:18.060704 30025 log_verifier.cc:126] Checking tablet 80471cfd4a944776baf05c4179f64a10
I20240502 21:26:18.070088 30025 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 80471cfd4a944776baf05c4179f64a10
I20240502 21:26:18.070161 30025 log_verifier.cc:126] Checking tablet 9812cd5d31154807a4f605eb615553f1
I20240502 21:26:18.075354 30025 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 9812cd5d31154807a4f605eb615553f1
I20240502 21:26:18.075423 30025 log_verifier.cc:126] Checking tablet 9873952de1224793b8b7f17ef92295b7
I20240502 21:26:18.084980 30025 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 9873952de1224793b8b7f17ef92295b7
I20240502 21:26:18.085049 30025 log_verifier.cc:126] Checking tablet a80b4f90d4b04e1886e494cf23a6284b
I20240502 21:26:18.094425 30025 log_verifier.cc:177] Verified matching terms for 6 ops in tablet a80b4f90d4b04e1886e494cf23a6284b
I20240502 21:26:18.094498 30025 log_verifier.cc:126] Checking tablet bc9b263fc4224e8eb8ca3c82a808bce8
I20240502 21:26:18.103889 30025 log_verifier.cc:177] Verified matching terms for 3 ops in tablet bc9b263fc4224e8eb8ca3c82a808bce8
I20240502 21:26:18.103955 30025 log_verifier.cc:126] Checking tablet cc8ba66c2b954876891bdded155d2b91
I20240502 21:26:18.112768 30025 log_verifier.cc:177] Verified matching terms for 3 ops in tablet cc8ba66c2b954876891bdded155d2b91
I20240502 21:26:18.112836 30025 log_verifier.cc:126] Checking tablet cd16f492f22c478a946d1da05f6a91b4
I20240502 21:26:18.122159 30025 log_verifier.cc:177] Verified matching terms for 6 ops in tablet cd16f492f22c478a946d1da05f6a91b4
I20240502 21:26:18.122242 30025 log_verifier.cc:126] Checking tablet d484c2aeb0904564997462a634b541d3
I20240502 21:26:18.131743 30025 log_verifier.cc:177] Verified matching terms for 6 ops in tablet d484c2aeb0904564997462a634b541d3
I20240502 21:26:18.131815 30025 log_verifier.cc:126] Checking tablet e60fc6b2f53444869f20be78f7c7cd6e
I20240502 21:26:18.143329 30025 log_verifier.cc:177] Verified matching terms for 6 ops in tablet e60fc6b2f53444869f20be78f7c7cd6e
I20240502 21:26:18.144598 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 32043
I20240502 21:26:18.145913 32167 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:18.360463 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 32043
I20240502 21:26:18.401789 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 32177
I20240502 21:26:18.402832 32302 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:18.618242 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 32177
W20240502 21:26:18.654531 31967 connection.cc:535] server connection from 127.29.82.66:46049 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:26:18.655181 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 828
I20240502 21:26:18.661484 952 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:18.860482 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 828
I20240502 21:26:18.892427 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 962
I20240502 21:26:18.898761 1086 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:19.089390 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 962
I20240502 21:26:19.120937 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 1094
I20240502 21:26:19.127211 1217 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:19.314967 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 1094
I20240502 21:26:19.346194 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 1226
I20240502 21:26:19.352349 1349 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:19.540935 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 1226
I20240502 21:26:19.571843 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 1357
I20240502 21:26:19.578095 1480 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:19.773170 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 1357
I20240502 21:26:19.804919 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 3547
I20240502 21:26:19.811058 3669 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:19.930163 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 3547
I20240502 21:26:19.951594 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 31948
I20240502 21:26:19.952651 32012 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:20.202023 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 31948
2024-05-02T21:26:20Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/0 (44850 ms)
[----------] 1 test from RebalanceParamTest (44850 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/0
I20240502 21:26:20.286569 30025 test_util.cc:268] Using random seed: 1091737170
I20240502 21:26:20.288576 30025 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:26:20.288637 30025 ts_itest-base.cc:113] --------------
I20240502 21:26:20.288693 30025 ts_itest-base.cc:114] 8 tablet servers
I20240502 21:26:20.288740 30025 ts_itest-base.cc:115] 3 replicas per TS
I20240502 21:26:20.288805 30025 ts_itest-base.cc:116] --------------
2024-05-02T21:26:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:26:20Z Disabled control of system clock
I20240502 21:26:20.321166 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.82.126:37445
--webserver_interface=127.29.82.126
--webserver_port=0
--builtin_ntp_servers=127.29.82.84:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.82.126:37445
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240502 21:26:20.651865 4269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:20.652148 4269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:20.652274 4269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:20.659319 4269 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:20.659476 4269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 21:26:20.659564 4269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:20.659675 4269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 21:26:20.659736 4269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 21:26:20.668151 4269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.29.82.126:37445
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.82.126:37445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.29.82.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:20.669422 4269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:20.671201 4269 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:20.683084 4274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:20.684032 4277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:20.684968 4275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:20.686321 4269 server_base.cc:1034] running on GCE node
I20240502 21:26:20.687083 4269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:20.688511 4269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:20.689707 4269 hybrid_clock.cc:648] HybridClock initialized: now 1714685180689663 us; error 36 us; skew 500 ppm
I20240502 21:26:20.690055 4269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:20.692324 4269 webserver.cc:460] Webserver started at http://127.29.82.126:39249/ using document root <none> and password file <none>
I20240502 21:26:20.693037 4269 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:20.693146 4269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:20.693451 4269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:20.697499 4269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e4b7d987f0ff4a3589447bd47c1ddd44"
format_stamp: "Formatted at 2024-05-02 21:26:20 on dist-test-slave-xzsm"
I20240502 21:26:20.698285 4269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e4b7d987f0ff4a3589447bd47c1ddd44"
format_stamp: "Formatted at 2024-05-02 21:26:20 on dist-test-slave-xzsm"
I20240502 21:26:20.703166 4269 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20240502 21:26:20.706624 4283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:20.707980 4269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:20.708149 4269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e4b7d987f0ff4a3589447bd47c1ddd44"
format_stamp: "Formatted at 2024-05-02 21:26:20 on dist-test-slave-xzsm"
I20240502 21:26:20.708359 4269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:20.732501 4269 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:20.733307 4269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:20.733570 4269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:20.751549 4269 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.126:37445
I20240502 21:26:20.751597 4335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.126:37445 every 8 connection(s)
I20240502 21:26:20.752874 4269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 21:26:20.756465 4336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:20.761288 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 4269
I20240502 21:26:20.761499 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 21:26:20.766604 4336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Bootstrap starting.
I20240502 21:26:20.769995 4336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:20.771253 4336 log.cc:824] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:20.774726 4336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: No bootstrap required, opened a new log
I20240502 21:26:20.780614 4336 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b7d987f0ff4a3589447bd47c1ddd44" member_type: VOTER last_known_addr { host: "127.29.82.126" port: 37445 } }
I20240502 21:26:20.780964 4336 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:20.781069 4336 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4b7d987f0ff4a3589447bd47c1ddd44, State: Initialized, Role: FOLLOWER
I20240502 21:26:20.781703 4336 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b7d987f0ff4a3589447bd47c1ddd44" member_type: VOTER last_known_addr { host: "127.29.82.126" port: 37445 } }
I20240502 21:26:20.781908 4336 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:26:20.782037 4336 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:26:20.782195 4336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:20.784312 4336 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b7d987f0ff4a3589447bd47c1ddd44" member_type: VOTER last_known_addr { host: "127.29.82.126" port: 37445 } }
I20240502 21:26:20.784765 4336 leader_election.cc:302] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4b7d987f0ff4a3589447bd47c1ddd44; no voters:
I20240502 21:26:20.785336 4336 leader_election.cc:288] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:26:20.785465 4341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:20.786717 4341 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44 [term 1 LEADER]: Becoming Leader. State: Replica: e4b7d987f0ff4a3589447bd47c1ddd44, State: Running, Role: LEADER
I20240502 21:26:20.787277 4341 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b7d987f0ff4a3589447bd47c1ddd44" member_type: VOTER last_known_addr { host: "127.29.82.126" port: 37445 } }
I20240502 21:26:20.787432 4336 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:26:20.790355 4343 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4b7d987f0ff4a3589447bd47c1ddd44. Latest consensus state: current_term: 1 leader_uuid: "e4b7d987f0ff4a3589447bd47c1ddd44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b7d987f0ff4a3589447bd47c1ddd44" member_type: VOTER last_known_addr { host: "127.29.82.126" port: 37445 } } }
I20240502 21:26:20.790599 4342 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4b7d987f0ff4a3589447bd47c1ddd44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b7d987f0ff4a3589447bd47c1ddd44" member_type: VOTER last_known_addr { host: "127.29.82.126" port: 37445 } } }
I20240502 21:26:20.790771 4343 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44 [sys.catalog]: This master's current role is: LEADER
I20240502 21:26:20.790858 4342 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44 [sys.catalog]: This master's current role is: LEADER
I20240502 21:26:20.791666 4348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:26:20.796949 4348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:26:20.805928 4348 catalog_manager.cc:1347] Generated new cluster ID: 6d5cd7b0d6f24619895ae2aeec209481
I20240502 21:26:20.806032 4348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:26:20.818861 4348 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:26:20.820037 4348 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:26:20.829077 4348 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Generated new TSK 0
I20240502 21:26:20.829820 4348 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:26:20.838160 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.65:0
--local_ip_for_outbound_sockets=127.29.82.65
--webserver_interface=127.29.82.65
--webserver_port=0
--tserver_master_addrs=127.29.82.126:37445
--builtin_ntp_servers=127.29.82.84:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 21:26:21.168059 4360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:21.168341 4360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:21.168485 4360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:21.175942 4360 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:21.176093 4360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:21.176259 4360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.65
I20240502 21:26:21.185176 4360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.29.82.65
--webserver_port=0
--tserver_master_addrs=127.29.82.126:37445
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.65
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:21.186607 4360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:21.188560 4360 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:21.201530 4365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:21.202761 4366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:21.203753 4368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:21.204751 4360 server_base.cc:1034] running on GCE node
I20240502 21:26:21.205317 4360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:21.206293 4360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:21.207509 4360 hybrid_clock.cc:648] HybridClock initialized: now 1714685181207461 us; error 40 us; skew 500 ppm
I20240502 21:26:21.207842 4360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:21.209949 4360 webserver.cc:460] Webserver started at http://127.29.82.65:44967/ using document root <none> and password file <none>
I20240502 21:26:21.210709 4360 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:21.210832 4360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:21.211144 4360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:21.213864 4360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "71e1375b799b439fa3c83ca763c039c1"
format_stamp: "Formatted at 2024-05-02 21:26:21 on dist-test-slave-xzsm"
I20240502 21:26:21.214633 4360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "71e1375b799b439fa3c83ca763c039c1"
format_stamp: "Formatted at 2024-05-02 21:26:21 on dist-test-slave-xzsm"
I20240502 21:26:21.219643 4360 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20240502 21:26:21.222959 4374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:21.224261 4360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:21.224419 4360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "71e1375b799b439fa3c83ca763c039c1"
format_stamp: "Formatted at 2024-05-02 21:26:21 on dist-test-slave-xzsm"
I20240502 21:26:21.224607 4360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:21.244774 4360 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:21.245553 4360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:21.245821 4360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:21.246800 4360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:21.248255 4360 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:21.248361 4360 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:21.248484 4360 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:21.248571 4360 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:21.280419 4360 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.65:42781
I20240502 21:26:21.280468 4488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.65:42781 every 8 connection(s)
I20240502 21:26:21.281718 4360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 21:26:21.291772 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 4360
I20240502 21:26:21.292042 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 21:26:21.294603 4489 heartbeater.cc:347] Connected to a master server at 127.29.82.126:37445
I20240502 21:26:21.294907 4489 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:21.295382 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.66:0
--local_ip_for_outbound_sockets=127.29.82.66
--webserver_interface=127.29.82.66
--webserver_port=0
--tserver_master_addrs=127.29.82.126:37445
--builtin_ntp_servers=127.29.82.84:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:26:21.295758 4489 heartbeater.cc:511] Master 127.29.82.126:37445 requested a full tablet report, sending...
I20240502 21:26:21.297706 4299 ts_manager.cc:194] Registered new tserver with Master: 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:21.299214 4299 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.65:36645
W20240502 21:26:21.623749 4493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:21.624035 4493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:21.624210 4493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:21.631687 4493 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:21.631799 4493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:21.631968 4493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.66
I20240502 21:26:21.640738 4493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.29.82.66
--webserver_port=0
--tserver_master_addrs=127.29.82.126:37445
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.66
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:21.642069 4493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:21.644132 4493 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:21.657577 4498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:21.658478 4501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:21.659538 4499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:21.660954 4493 server_base.cc:1034] running on GCE node
I20240502 21:26:21.661561 4493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:21.662503 4493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:21.663658 4493 hybrid_clock.cc:648] HybridClock initialized: now 1714685181663610 us; error 55 us; skew 500 ppm
I20240502 21:26:21.663982 4493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:21.666328 4493 webserver.cc:460] Webserver started at http://127.29.82.66:40515/ using document root <none> and password file <none>
I20240502 21:26:21.667130 4493 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:21.667265 4493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:21.667642 4493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:21.670408 4493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
format_stamp: "Formatted at 2024-05-02 21:26:21 on dist-test-slave-xzsm"
I20240502 21:26:21.671193 4493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
format_stamp: "Formatted at 2024-05-02 21:26:21 on dist-test-slave-xzsm"
I20240502 21:26:21.676525 4493 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240502 21:26:21.680153 4507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:21.681493 4493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20240502 21:26:21.681689 4493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
format_stamp: "Formatted at 2024-05-02 21:26:21 on dist-test-slave-xzsm"
I20240502 21:26:21.681882 4493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:21.698304 4493 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:21.699116 4493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:21.699383 4493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:21.700340 4493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:21.701792 4493 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:21.701895 4493 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:21.702011 4493 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:21.702085 4493 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:21.734439 4493 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.66:35781
I20240502 21:26:21.734493 4620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.66:35781 every 8 connection(s)
I20240502 21:26:21.735749 4493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 21:26:21.736418 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 4493
I20240502 21:26:21.736629 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 21:26:21.740406 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.67:0
--local_ip_for_outbound_sockets=127.29.82.67
--webserver_interface=127.29.82.67
--webserver_port=0
--tserver_master_addrs=127.29.82.126:37445
--builtin_ntp_servers=127.29.82.84:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:26:21.750072 4621 heartbeater.cc:347] Connected to a master server at 127.29.82.126:37445
I20240502 21:26:21.750368 4621 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:21.751236 4621 heartbeater.cc:511] Master 127.29.82.126:37445 requested a full tablet report, sending...
I20240502 21:26:21.752925 4299 ts_manager.cc:194] Registered new tserver with Master: 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:21.754061 4299 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.66:45603
W20240502 21:26:22.070765 4624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:22.071053 4624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:22.071142 4624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:22.079174 4624 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:22.079299 4624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:22.079424 4624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.67
I20240502 21:26:22.089114 4624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.29.82.67
--webserver_port=0
--tserver_master_addrs=127.29.82.126:37445
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.67
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:22.090462 4624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:22.092257 4624 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:22.105319 4629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:22.106527 4630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:22.107439 4632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:22.108574 4624 server_base.cc:1034] running on GCE node
I20240502 21:26:22.109251 4624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:22.110297 4624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:22.111517 4624 hybrid_clock.cc:648] HybridClock initialized: now 1714685182111464 us; error 41 us; skew 500 ppm
I20240502 21:26:22.111892 4624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:22.114368 4624 webserver.cc:460] Webserver started at http://127.29.82.67:34411/ using document root <none> and password file <none>
I20240502 21:26:22.115164 4624 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:22.115293 4624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:22.115675 4624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:22.118373 4624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "398c3c8f3195492c900439ad5e1931bb"
format_stamp: "Formatted at 2024-05-02 21:26:22 on dist-test-slave-xzsm"
I20240502 21:26:22.119199 4624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "398c3c8f3195492c900439ad5e1931bb"
format_stamp: "Formatted at 2024-05-02 21:26:22 on dist-test-slave-xzsm"
I20240502 21:26:22.124698 4624 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240502 21:26:22.128198 4638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:22.129737 4624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:26:22.129904 4624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "398c3c8f3195492c900439ad5e1931bb"
format_stamp: "Formatted at 2024-05-02 21:26:22 on dist-test-slave-xzsm"
I20240502 21:26:22.130111 4624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:22.168995 4624 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:22.170137 4624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:22.170469 4624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:22.171680 4624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:22.173348 4624 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:22.173452 4624 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:22.173568 4624 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:22.173650 4624 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:22.207691 4624 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.67:34537
I20240502 21:26:22.207762 4751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.67:34537 every 8 connection(s)
I20240502 21:26:22.209053 4624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:26:22.215627 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 4624
I20240502 21:26:22.215857 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 21:26:22.219539 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.68:0
--local_ip_for_outbound_sockets=127.29.82.68
--webserver_interface=127.29.82.68
--webserver_port=0
--tserver_master_addrs=127.29.82.126:37445
--builtin_ntp_servers=127.29.82.84:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:26:22.223248 4752 heartbeater.cc:347] Connected to a master server at 127.29.82.126:37445
I20240502 21:26:22.223629 4752 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:22.224571 4752 heartbeater.cc:511] Master 127.29.82.126:37445 requested a full tablet report, sending...
I20240502 21:26:22.226119 4299 ts_manager.cc:194] Registered new tserver with Master: 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:22.227306 4299 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.67:47293
I20240502 21:26:22.301838 4489 heartbeater.cc:503] Master 127.29.82.126:37445 was elected leader, sending a full tablet report...
W20240502 21:26:22.565054 4755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:22.565454 4755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:22.565668 4755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:22.573369 4755 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:22.573488 4755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:22.573668 4755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.68
I20240502 21:26:22.582855 4755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.29.82.68
--webserver_port=0
--tserver_master_addrs=127.29.82.126:37445
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.68
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:22.584347 4755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:22.586233 4755 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:22.599433 4760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:22.600389 4761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:22.601240 4763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:22.602608 4755 server_base.cc:1034] running on GCE node
I20240502 21:26:22.603169 4755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:22.604156 4755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:22.605363 4755 hybrid_clock.cc:648] HybridClock initialized: now 1714685182605323 us; error 34 us; skew 500 ppm
I20240502 21:26:22.605706 4755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:22.608124 4755 webserver.cc:460] Webserver started at http://127.29.82.68:34459/ using document root <none> and password file <none>
I20240502 21:26:22.608860 4755 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:22.608997 4755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:22.609305 4755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:22.612057 4755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "73a1f2ca1f544de9b0e3b37015f85328"
format_stamp: "Formatted at 2024-05-02 21:26:22 on dist-test-slave-xzsm"
I20240502 21:26:22.612840 4755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "73a1f2ca1f544de9b0e3b37015f85328"
format_stamp: "Formatted at 2024-05-02 21:26:22 on dist-test-slave-xzsm"
I20240502 21:26:22.617993 4755 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 21:26:22.621426 4769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:22.622845 4755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:22.623034 4755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "73a1f2ca1f544de9b0e3b37015f85328"
format_stamp: "Formatted at 2024-05-02 21:26:22 on dist-test-slave-xzsm"
I20240502 21:26:22.623255 4755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:22.638612 4755 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:22.639416 4755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:22.639768 4755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:22.640655 4755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:22.642079 4755 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:22.642206 4755 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:22.642316 4755 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:22.642395 4755 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:22.675158 4755 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.68:42449
I20240502 21:26:22.675200 4883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.68:42449 every 8 connection(s)
I20240502 21:26:22.676573 4755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:26:22.682183 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 4755
I20240502 21:26:22.682379 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 21:26:22.685863 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.69:0
--local_ip_for_outbound_sockets=127.29.82.69
--webserver_interface=127.29.82.69
--webserver_port=0
--tserver_master_addrs=127.29.82.126:37445
--builtin_ntp_servers=127.29.82.84:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:26:22.690416 4884 heartbeater.cc:347] Connected to a master server at 127.29.82.126:37445
I20240502 21:26:22.690739 4884 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:22.691604 4884 heartbeater.cc:511] Master 127.29.82.126:37445 requested a full tablet report, sending...
I20240502 21:26:22.693038 4299 ts_manager.cc:194] Registered new tserver with Master: 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:22.693807 4299 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.68:58995
I20240502 21:26:22.756960 4621 heartbeater.cc:503] Master 127.29.82.126:37445 was elected leader, sending a full tablet report...
W20240502 21:26:23.017552 4887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:23.017885 4887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:23.018039 4887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:23.025496 4887 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:23.025648 4887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:23.025823 4887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.69
I20240502 21:26:23.035248 4887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.29.82.69
--webserver_port=0
--tserver_master_addrs=127.29.82.126:37445
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.69
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:23.036652 4887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:23.038395 4887 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:23.050940 4895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:23.051863 4893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:23.053177 4892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:23.054304 4887 server_base.cc:1034] running on GCE node
I20240502 21:26:23.054961 4887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:23.055891 4887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:23.057085 4887 hybrid_clock.cc:648] HybridClock initialized: now 1714685183057039 us; error 39 us; skew 500 ppm
I20240502 21:26:23.057394 4887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:23.064057 4887 webserver.cc:460] Webserver started at http://127.29.82.69:40947/ using document root <none> and password file <none>
I20240502 21:26:23.064926 4887 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:23.065044 4887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:23.065366 4887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:23.068061 4887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "39981a8dfe94462b9a73ba489af4c714"
format_stamp: "Formatted at 2024-05-02 21:26:23 on dist-test-slave-xzsm"
I20240502 21:26:23.068822 4887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "39981a8dfe94462b9a73ba489af4c714"
format_stamp: "Formatted at 2024-05-02 21:26:23 on dist-test-slave-xzsm"
I20240502 21:26:23.073879 4887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 21:26:23.077332 4902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:23.078660 4887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 21:26:23.078831 4887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "39981a8dfe94462b9a73ba489af4c714"
format_stamp: "Formatted at 2024-05-02 21:26:23 on dist-test-slave-xzsm"
I20240502 21:26:23.079034 4887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:23.097134 4887 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:23.098073 4887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:23.098356 4887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:23.099324 4887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:23.100870 4887 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:23.100972 4887 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:23.101094 4887 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:23.101169 4887 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:23.134315 4887 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.69:33193
I20240502 21:26:23.134361 5015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.69:33193 every 8 connection(s)
I20240502 21:26:23.135820 4887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:26:23.138556 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 4887
I20240502 21:26:23.138767 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 21:26:23.141772 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.70:0
--local_ip_for_outbound_sockets=127.29.82.70
--webserver_interface=127.29.82.70
--webserver_port=0
--tserver_master_addrs=127.29.82.126:37445
--builtin_ntp_servers=127.29.82.84:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:26:23.149060 5016 heartbeater.cc:347] Connected to a master server at 127.29.82.126:37445
I20240502 21:26:23.149348 5016 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:23.150077 5016 heartbeater.cc:511] Master 127.29.82.126:37445 requested a full tablet report, sending...
I20240502 21:26:23.151486 4299 ts_manager.cc:194] Registered new tserver with Master: 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:26:23.152613 4299 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.69:38501
I20240502 21:26:23.229991 4752 heartbeater.cc:503] Master 127.29.82.126:37445 was elected leader, sending a full tablet report...
W20240502 21:26:23.478075 5019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:23.478361 5019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:23.478463 5019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:23.485872 5019 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:23.485997 5019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:23.486124 5019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.70
I20240502 21:26:23.495608 5019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.29.82.70
--webserver_port=0
--tserver_master_addrs=127.29.82.126:37445
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.70
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:23.497001 5019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:23.498731 5019 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:23.512449 5025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:23.513289 5024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:23.514166 5027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:23.515704 5019 server_base.cc:1034] running on GCE node
I20240502 21:26:23.516324 5019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:23.517266 5019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:23.518487 5019 hybrid_clock.cc:648] HybridClock initialized: now 1714685183518440 us; error 36 us; skew 500 ppm
I20240502 21:26:23.518828 5019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:23.521086 5019 webserver.cc:460] Webserver started at http://127.29.82.70:34273/ using document root <none> and password file <none>
I20240502 21:26:23.521844 5019 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:23.521944 5019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:23.522279 5019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:23.524960 5019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "d0456c5e55a542128688904e003d403c"
format_stamp: "Formatted at 2024-05-02 21:26:23 on dist-test-slave-xzsm"
I20240502 21:26:23.525707 5019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "d0456c5e55a542128688904e003d403c"
format_stamp: "Formatted at 2024-05-02 21:26:23 on dist-test-slave-xzsm"
I20240502 21:26:23.531160 5019 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240502 21:26:23.534404 5033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:23.535821 5019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:23.535970 5019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "d0456c5e55a542128688904e003d403c"
format_stamp: "Formatted at 2024-05-02 21:26:23 on dist-test-slave-xzsm"
I20240502 21:26:23.536164 5019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:23.569204 5019 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:23.570168 5019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:23.570461 5019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:23.571458 5019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:23.573029 5019 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:23.573125 5019 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:23.573248 5019 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:23.573321 5019 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:23.606163 5019 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.70:40719
I20240502 21:26:23.606269 5146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.70:40719 every 8 connection(s)
I20240502 21:26:23.607520 5019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 21:26:23.614284 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 5019
I20240502 21:26:23.614440 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 21:26:23.617810 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.71:0
--local_ip_for_outbound_sockets=127.29.82.71
--webserver_interface=127.29.82.71
--webserver_port=0
--tserver_master_addrs=127.29.82.126:37445
--builtin_ntp_servers=127.29.82.84:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:26:23.620465 5147 heartbeater.cc:347] Connected to a master server at 127.29.82.126:37445
I20240502 21:26:23.620801 5147 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:23.621727 5147 heartbeater.cc:511] Master 127.29.82.126:37445 requested a full tablet report, sending...
I20240502 21:26:23.623194 4299 ts_manager.cc:194] Registered new tserver with Master: d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:26:23.624011 4299 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.70:42273
I20240502 21:26:23.696378 4884 heartbeater.cc:503] Master 127.29.82.126:37445 was elected leader, sending a full tablet report...
W20240502 21:26:23.954803 5150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:23.955058 5150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:23.955168 5150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:23.962404 5150 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:23.962517 5150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:23.962637 5150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.71
I20240502 21:26:23.971311 5150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.29.82.71
--webserver_port=0
--tserver_master_addrs=127.29.82.126:37445
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.71
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:23.972635 5150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:23.974318 5150 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:23.987406 5155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:23.988572 5156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:23.989617 5158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:23.990702 5150 server_base.cc:1034] running on GCE node
I20240502 21:26:23.991297 5150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:23.992317 5150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:23.993520 5150 hybrid_clock.cc:648] HybridClock initialized: now 1714685183993467 us; error 43 us; skew 500 ppm
I20240502 21:26:23.993841 5150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:23.996206 5150 webserver.cc:460] Webserver started at http://127.29.82.71:46521/ using document root <none> and password file <none>
I20240502 21:26:23.997025 5150 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:23.997128 5150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:23.997400 5150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:23.999981 5150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "31a72286397c4edc9a63eca3412be168"
format_stamp: "Formatted at 2024-05-02 21:26:23 on dist-test-slave-xzsm"
I20240502 21:26:24.000694 5150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "31a72286397c4edc9a63eca3412be168"
format_stamp: "Formatted at 2024-05-02 21:26:23 on dist-test-slave-xzsm"
I20240502 21:26:24.005511 5150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 21:26:24.008631 5164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:24.009920 5150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:24.010072 5150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "31a72286397c4edc9a63eca3412be168"
format_stamp: "Formatted at 2024-05-02 21:26:23 on dist-test-slave-xzsm"
I20240502 21:26:24.010249 5150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:24.028654 5150 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:24.029484 5150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:24.029786 5150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:24.030699 5150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:24.032233 5150 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:24.032333 5150 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:24.032449 5150 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:24.032526 5150 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:24.064077 5150 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.71:33913
I20240502 21:26:24.064150 5277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.71:33913 every 8 connection(s)
I20240502 21:26:24.065629 5150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 21:26:24.070333 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 5150
I20240502 21:26:24.070540 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 21:26:24.073704 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.72:0
--local_ip_for_outbound_sockets=127.29.82.72
--webserver_interface=127.29.82.72
--webserver_port=0
--tserver_master_addrs=127.29.82.126:37445
--builtin_ntp_servers=127.29.82.84:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:26:24.079118 5278 heartbeater.cc:347] Connected to a master server at 127.29.82.126:37445
I20240502 21:26:24.079418 5278 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:24.080191 5278 heartbeater.cc:511] Master 127.29.82.126:37445 requested a full tablet report, sending...
I20240502 21:26:24.081552 4299 ts_manager.cc:194] Registered new tserver with Master: 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:26:24.082374 4299 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.71:54599
I20240502 21:26:24.154929 5016 heartbeater.cc:503] Master 127.29.82.126:37445 was elected leader, sending a full tablet report...
W20240502 21:26:24.412858 5281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:24.413168 5281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:24.413322 5281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:24.421011 5281 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:24.421154 5281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:24.421315 5281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.72
I20240502 21:26:24.430599 5281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.29.82.72
--webserver_port=0
--tserver_master_addrs=127.29.82.126:37445
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.72
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:24.431946 5281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:24.433733 5281 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:24.447135 5286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:24.447952 5287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:24.448850 5289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:24.450291 5281 server_base.cc:1034] running on GCE node
I20240502 21:26:24.450881 5281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:24.451783 5281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:24.452986 5281 hybrid_clock.cc:648] HybridClock initialized: now 1714685184452941 us; error 39 us; skew 500 ppm
I20240502 21:26:24.453320 5281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:24.455559 5281 webserver.cc:460] Webserver started at http://127.29.82.72:37911/ using document root <none> and password file <none>
I20240502 21:26:24.456317 5281 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:24.456422 5281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:24.456739 5281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:24.459277 5281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "0ddd486109f5434cace13d60d633ac04"
format_stamp: "Formatted at 2024-05-02 21:26:24 on dist-test-slave-xzsm"
I20240502 21:26:24.460045 5281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "0ddd486109f5434cace13d60d633ac04"
format_stamp: "Formatted at 2024-05-02 21:26:24 on dist-test-slave-xzsm"
I20240502 21:26:24.464883 5281 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240502 21:26:24.468266 5295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:24.469590 5281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:26:24.469770 5281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "0ddd486109f5434cace13d60d633ac04"
format_stamp: "Formatted at 2024-05-02 21:26:24 on dist-test-slave-xzsm"
I20240502 21:26:24.469970 5281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:24.481820 5281 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:24.482630 5281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:24.482911 5281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:24.483814 5281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:24.485348 5281 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:24.485436 5281 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:24.485545 5281 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:24.485618 5281 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:24.517838 5281 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.72:41223
I20240502 21:26:24.517889 5409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.72:41223 every 8 connection(s)
I20240502 21:26:24.519156 5281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 21:26:24.526858 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 5281
I20240502 21:26:24.527029 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 21:26:24.532011 5410 heartbeater.cc:347] Connected to a master server at 127.29.82.126:37445
I20240502 21:26:24.532289 5410 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:24.532976 5410 heartbeater.cc:511] Master 127.29.82.126:37445 requested a full tablet report, sending...
I20240502 21:26:24.534238 4299 ts_manager.cc:194] Registered new tserver with Master: 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:26:24.535087 4299 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.72:50419
I20240502 21:26:24.543448 30025 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 21:26:24.559341 30025 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 21:26:24.559484 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 4887
I20240502 21:26:24.567188 5011 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:24.626406 5147 heartbeater.cc:503] Master 127.29.82.126:37445 was elected leader, sending a full tablet report...
I20240502 21:26:24.650422 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 4887
I20240502 21:26:24.670258 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 5019
I20240502 21:26:24.680300 5142 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:24.763262 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 5019
I20240502 21:26:24.780700 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 5150
I20240502 21:26:24.788616 5273 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:24.867992 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 5150
I20240502 21:26:24.885664 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 5281
I20240502 21:26:24.894291 5405 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:24.974189 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 5281
I20240502 21:26:28.744225 4299 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:28.780895 4423 tablet_service.cc:1467] Processing CreateTablet for tablet 25ce93e8577248a4b915265fb781faae (DEFAULT_TABLE table=rebalance_test_table_0 [id=927c4d77ce3e435da867fbd206041c89]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:28.780781 4422 tablet_service.cc:1467] Processing CreateTablet for tablet 7feb601648e14a998cfc7331a6934ac1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=927c4d77ce3e435da867fbd206041c89]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:28.781679 4421 tablet_service.cc:1467] Processing CreateTablet for tablet 23c09be2916844a9b9bb5855d3566b86 (DEFAULT_TABLE table=rebalance_test_table_0 [id=927c4d77ce3e435da867fbd206041c89]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:28.783597 4423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25ce93e8577248a4b915265fb781faae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:28.783088 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 25ce93e8577248a4b915265fb781faae (DEFAULT_TABLE table=rebalance_test_table_0 [id=927c4d77ce3e435da867fbd206041c89]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:28.784099 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23c09be2916844a9b9bb5855d3566b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:28.784617 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25ce93e8577248a4b915265fb781faae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:28.785975 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7feb601648e14a998cfc7331a6934ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:28.786929 4685 tablet_service.cc:1467] Processing CreateTablet for tablet 7feb601648e14a998cfc7331a6934ac1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=927c4d77ce3e435da867fbd206041c89]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:28.787420 4685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7feb601648e14a998cfc7331a6934ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:28.787652 4818 tablet_service.cc:1467] Processing CreateTablet for tablet 25ce93e8577248a4b915265fb781faae (DEFAULT_TABLE table=rebalance_test_table_0 [id=927c4d77ce3e435da867fbd206041c89]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:28.789315 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25ce93e8577248a4b915265fb781faae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:28.795662 4817 tablet_service.cc:1467] Processing CreateTablet for tablet 23c09be2916844a9b9bb5855d3566b86 (DEFAULT_TABLE table=rebalance_test_table_0 [id=927c4d77ce3e435da867fbd206041c89]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:28.796269 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23c09be2916844a9b9bb5855d3566b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:28.803973 4554 tablet_service.cc:1467] Processing CreateTablet for tablet 23c09be2916844a9b9bb5855d3566b86 (DEFAULT_TABLE table=rebalance_test_table_0 [id=927c4d77ce3e435da867fbd206041c89]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:28.805079 5438 tablet_bootstrap.cc:492] T 7feb601648e14a998cfc7331a6934ac1 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:28.805320 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23c09be2916844a9b9bb5855d3566b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:28.806499 5439 tablet_bootstrap.cc:492] T 25ce93e8577248a4b915265fb781faae P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:28.808807 5438 tablet_bootstrap.cc:654] T 7feb601648e14a998cfc7331a6934ac1 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:28.803251 4555 tablet_service.cc:1467] Processing CreateTablet for tablet 7feb601648e14a998cfc7331a6934ac1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=927c4d77ce3e435da867fbd206041c89]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:28.809403 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7feb601648e14a998cfc7331a6934ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:28.810024 5439 tablet_bootstrap.cc:654] T 25ce93e8577248a4b915265fb781faae P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:28.811367 5438 log.cc:824] T 7feb601648e14a998cfc7331a6934ac1 P 398c3c8f3195492c900439ad5e1931bb: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:28.811376 5439 log.cc:824] T 25ce93e8577248a4b915265fb781faae P 73a1f2ca1f544de9b0e3b37015f85328: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:28.824280 5439 tablet_bootstrap.cc:492] T 25ce93e8577248a4b915265fb781faae P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:28.824270 5438 tablet_bootstrap.cc:492] T 7feb601648e14a998cfc7331a6934ac1 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:28.824591 5438 ts_tablet_manager.cc:1367] T 7feb601648e14a998cfc7331a6934ac1 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.005s
I20240502 21:26:28.824616 5439 ts_tablet_manager.cc:1367] T 25ce93e8577248a4b915265fb781faae P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.018s user 0.014s sys 0.004s
I20240502 21:26:28.827867 5442 tablet_bootstrap.cc:492] T 23c09be2916844a9b9bb5855d3566b86 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:28.834021 5442 tablet_bootstrap.cc:654] T 23c09be2916844a9b9bb5855d3566b86 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:28.836472 5443 tablet_bootstrap.cc:492] T 23c09be2916844a9b9bb5855d3566b86 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:28.836412 5439 raft_consensus.cc:348] T 25ce93e8577248a4b915265fb781faae P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:28.836817 5439 raft_consensus.cc:374] T 25ce93e8577248a4b915265fb781faae P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:28.836920 5439 raft_consensus.cc:729] T 25ce93e8577248a4b915265fb781faae P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:28.837456 5442 log.cc:824] T 23c09be2916844a9b9bb5855d3566b86 P 7a736a8ebb164ce0a051c0fccd76c3b8: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:28.837702 5439 consensus_queue.cc:260] T 25ce93e8577248a4b915265fb781faae P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:28.839139 5438 raft_consensus.cc:348] T 7feb601648e14a998cfc7331a6934ac1 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:28.839704 5438 raft_consensus.cc:374] T 7feb601648e14a998cfc7331a6934ac1 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:28.839841 5438 raft_consensus.cc:729] T 7feb601648e14a998cfc7331a6934ac1 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:28.840294 5443 tablet_bootstrap.cc:654] T 23c09be2916844a9b9bb5855d3566b86 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:28.840694 5438 consensus_queue.cc:260] T 7feb601648e14a998cfc7331a6934ac1 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:28.841711 5443 log.cc:824] T 23c09be2916844a9b9bb5855d3566b86 P 71e1375b799b439fa3c83ca763c039c1: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:28.844332 5438 ts_tablet_manager.cc:1398] T 7feb601648e14a998cfc7331a6934ac1 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.020s user 0.014s sys 0.000s
I20240502 21:26:28.845136 5438 tablet_bootstrap.cc:492] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:28.847239 5438 tablet_bootstrap.cc:654] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:28.851249 5439 ts_tablet_manager.cc:1398] T 25ce93e8577248a4b915265fb781faae P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.026s user 0.013s sys 0.008s
I20240502 21:26:28.851742 5439 tablet_bootstrap.cc:492] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:28.853493 5439 tablet_bootstrap.cc:654] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:28.853684 5438 tablet_bootstrap.cc:492] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:28.853839 5438 ts_tablet_manager.cc:1367] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20240502 21:26:28.854981 5438 raft_consensus.cc:348] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:28.855259 5438 raft_consensus.cc:374] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:28.855379 5438 raft_consensus.cc:729] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:28.855780 5438 consensus_queue.cc:260] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:28.856348 5443 tablet_bootstrap.cc:492] T 23c09be2916844a9b9bb5855d3566b86 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:28.856693 5443 ts_tablet_manager.cc:1367] T 23c09be2916844a9b9bb5855d3566b86 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.004s
I20240502 21:26:28.857825 5442 tablet_bootstrap.cc:492] T 23c09be2916844a9b9bb5855d3566b86 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:28.858143 5442 ts_tablet_manager.cc:1367] T 23c09be2916844a9b9bb5855d3566b86 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.031s user 0.014s sys 0.000s
I20240502 21:26:28.859966 5446 raft_consensus.cc:482] T 7feb601648e14a998cfc7331a6934ac1 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:28.860239 5446 raft_consensus.cc:504] T 7feb601648e14a998cfc7331a6934ac1 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:28.861510 5439 tablet_bootstrap.cc:492] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:28.861661 5439 ts_tablet_manager.cc:1367] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20240502 21:26:28.862689 5439 raft_consensus.cc:348] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:28.862939 5439 raft_consensus.cc:374] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:28.863039 5439 raft_consensus.cc:729] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:28.863375 5439 consensus_queue.cc:260] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:28.864058 5439 ts_tablet_manager.cc:1398] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:26:28.864712 5446 leader_election.cc:288] T 7feb601648e14a998cfc7331a6934ac1 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:28.865314 5438 ts_tablet_manager.cc:1398] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240502 21:26:28.866789 5443 raft_consensus.cc:348] T 23c09be2916844a9b9bb5855d3566b86 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:28.867367 5443 raft_consensus.cc:374] T 23c09be2916844a9b9bb5855d3566b86 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:28.867535 5443 raft_consensus.cc:729] T 23c09be2916844a9b9bb5855d3566b86 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:28.868391 5443 consensus_queue.cc:260] T 23c09be2916844a9b9bb5855d3566b86 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:28.871507 5442 raft_consensus.cc:348] T 23c09be2916844a9b9bb5855d3566b86 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:28.872731 5442 raft_consensus.cc:374] T 23c09be2916844a9b9bb5855d3566b86 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:28.872840 5442 raft_consensus.cc:729] T 23c09be2916844a9b9bb5855d3566b86 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:28.873255 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7feb601648e14a998cfc7331a6934ac1" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:28.873627 5442 consensus_queue.cc:260] T 23c09be2916844a9b9bb5855d3566b86 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
W20240502 21:26:28.874300 4639 leader_election.cc:341] T 7feb601648e14a998cfc7331a6934ac1 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:28.876086 5442 ts_tablet_manager.cc:1398] T 23c09be2916844a9b9bb5855d3566b86 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.018s user 0.010s sys 0.005s
I20240502 21:26:28.876410 5443 ts_tablet_manager.cc:1398] T 23c09be2916844a9b9bb5855d3566b86 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.020s user 0.011s sys 0.002s
I20240502 21:26:28.876541 5442 tablet_bootstrap.cc:492] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:28.878355 5442 tablet_bootstrap.cc:654] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:28.880201 5443 tablet_bootstrap.cc:492] T 7feb601648e14a998cfc7331a6934ac1 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:28.881376 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7feb601648e14a998cfc7331a6934ac1" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:28.881594 5442 tablet_bootstrap.cc:492] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:28.881752 5442 ts_tablet_manager.cc:1367] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:26:28.882629 5443 tablet_bootstrap.cc:654] T 7feb601648e14a998cfc7331a6934ac1 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
W20240502 21:26:28.882623 4642 leader_election.cc:341] T 7feb601648e14a998cfc7331a6934ac1 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:28.882763 4642 leader_election.cc:302] T 7feb601648e14a998cfc7331a6934ac1 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb; no voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:26:28.882856 5442 raft_consensus.cc:348] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:28.883049 5442 raft_consensus.cc:374] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:28.883144 5442 raft_consensus.cc:729] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:28.883383 5442 consensus_queue.cc:260] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:28.883637 5450 raft_consensus.cc:2726] T 7feb601648e14a998cfc7331a6934ac1 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:26:28.884044 5442 ts_tablet_manager.cc:1398] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 21:26:28.884920 5443 tablet_bootstrap.cc:492] T 7feb601648e14a998cfc7331a6934ac1 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:28.885058 5443 ts_tablet_manager.cc:1367] T 7feb601648e14a998cfc7331a6934ac1 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:26:28.886019 5443 raft_consensus.cc:348] T 7feb601648e14a998cfc7331a6934ac1 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:28.886219 5443 raft_consensus.cc:374] T 7feb601648e14a998cfc7331a6934ac1 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:28.886322 5443 raft_consensus.cc:729] T 7feb601648e14a998cfc7331a6934ac1 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:28.886627 5443 consensus_queue.cc:260] T 7feb601648e14a998cfc7331a6934ac1 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:28.887200 5443 ts_tablet_manager.cc:1398] T 7feb601648e14a998cfc7331a6934ac1 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:28.887497 5443 tablet_bootstrap.cc:492] T 25ce93e8577248a4b915265fb781faae P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:28.889132 5443 tablet_bootstrap.cc:654] T 25ce93e8577248a4b915265fb781faae P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:28.891017 5443 tablet_bootstrap.cc:492] T 25ce93e8577248a4b915265fb781faae P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:28.891170 5443 ts_tablet_manager.cc:1367] T 25ce93e8577248a4b915265fb781faae P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:26:28.891745 5443 raft_consensus.cc:348] T 25ce93e8577248a4b915265fb781faae P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:28.891934 5443 raft_consensus.cc:374] T 25ce93e8577248a4b915265fb781faae P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:28.892010 5443 raft_consensus.cc:729] T 25ce93e8577248a4b915265fb781faae P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:28.892216 5443 consensus_queue.cc:260] T 25ce93e8577248a4b915265fb781faae P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:28.892652 5443 ts_tablet_manager.cc:1398] T 25ce93e8577248a4b915265fb781faae P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 21:26:28.893877 5444 raft_consensus.cc:482] T 25ce93e8577248a4b915265fb781faae P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:28.894133 5444 raft_consensus.cc:504] T 25ce93e8577248a4b915265fb781faae P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:28.896486 5444 leader_election.cc:288] T 25ce93e8577248a4b915265fb781faae P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:28.901589 5444 raft_consensus.cc:482] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:28.901796 5444 raft_consensus.cc:504] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:28.902541 5444 leader_election.cc:288] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:28.903380 5450 raft_consensus.cc:482] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:28.903769 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25ce93e8577248a4b915265fb781faae" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:28.903645 5450 raft_consensus.cc:504] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:28.904107 4443 raft_consensus.cc:2454] T 25ce93e8577248a4b915265fb781faae P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:28.904752 5450 leader_election.cc:288] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:28.904758 4770 leader_election.cc:302] T 25ce93e8577248a4b915265fb781faae P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:28.905217 5444 raft_consensus.cc:2781] T 25ce93e8577248a4b915265fb781faae P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:28.905110 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c09be2916844a9b9bb5855d3566b86" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:28.905355 5444 raft_consensus.cc:482] T 25ce93e8577248a4b915265fb781faae P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:28.905299 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25ce93e8577248a4b915265fb781faae" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:28.905480 5444 raft_consensus.cc:3037] T 25ce93e8577248a4b915265fb781faae P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:28.905457 4442 raft_consensus.cc:2454] T 23c09be2916844a9b9bb5855d3566b86 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:28.905536 4443 raft_consensus.cc:2454] T 25ce93e8577248a4b915265fb781faae P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:26:28.905972 4639 leader_election.cc:302] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:26:28.906299 5450 raft_consensus.cc:2781] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:28.906417 5450 raft_consensus.cc:482] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:28.906530 5450 raft_consensus.cc:3037] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:28.906904 4770 leader_election.cc:302] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:28.909695 5450 raft_consensus.cc:504] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:28.910245 5444 raft_consensus.cc:504] T 25ce93e8577248a4b915265fb781faae P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:28.911155 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25ce93e8577248a4b915265fb781faae" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:28.911392 4443 raft_consensus.cc:3037] T 25ce93e8577248a4b915265fb781faae P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:28.911489 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25ce93e8577248a4b915265fb781faae" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:28.912073 4706 raft_consensus.cc:2379] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 73a1f2ca1f544de9b0e3b37015f85328 in current term 1: Already voted for candidate 398c3c8f3195492c900439ad5e1931bb in this term.
I20240502 21:26:28.913677 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25ce93e8577248a4b915265fb781faae" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:28.913899 4706 raft_consensus.cc:2379] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 73a1f2ca1f544de9b0e3b37015f85328 in current term 1: Already voted for candidate 398c3c8f3195492c900439ad5e1931bb in this term.
I20240502 21:26:28.914402 5444 leader_election.cc:288] T 25ce93e8577248a4b915265fb781faae P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:28.914526 4443 raft_consensus.cc:2454] T 25ce93e8577248a4b915265fb781faae P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:26:28.914628 5444 raft_consensus.cc:2781] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:28.914755 5444 raft_consensus.cc:482] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:28.914877 5444 raft_consensus.cc:3037] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:28.914989 4639 leader_election.cc:302] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:26:28.915287 5446 raft_consensus.cc:2781] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:28.916615 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25ce93e8577248a4b915265fb781faae" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:26:28.917284 4839 raft_consensus.cc:2379] T 25ce93e8577248a4b915265fb781faae P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 398c3c8f3195492c900439ad5e1931bb in current term 1: Already voted for candidate 73a1f2ca1f544de9b0e3b37015f85328 in this term.
I20240502 21:26:28.919096 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25ce93e8577248a4b915265fb781faae" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:26:28.919335 4839 raft_consensus.cc:2379] T 25ce93e8577248a4b915265fb781faae P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 398c3c8f3195492c900439ad5e1931bb in current term 1: Already voted for candidate 73a1f2ca1f544de9b0e3b37015f85328 in this term.
I20240502 21:26:28.919893 5450 leader_election.cc:288] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:28.921000 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c09be2916844a9b9bb5855d3566b86" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:28.921480 4575 raft_consensus.cc:2454] T 23c09be2916844a9b9bb5855d3566b86 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:28.923645 5444 raft_consensus.cc:504] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:28.923926 5446 raft_consensus.cc:686] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Becoming Leader. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Running, Role: LEADER
I20240502 21:26:28.924513 5446 consensus_queue.cc:237] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:28.924789 5444 leader_election.cc:288] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:28.924785 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c09be2916844a9b9bb5855d3566b86" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:26:28.924991 4575 raft_consensus.cc:3037] T 23c09be2916844a9b9bb5855d3566b86 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:28.925508 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c09be2916844a9b9bb5855d3566b86" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:28.925704 4442 raft_consensus.cc:3037] T 23c09be2916844a9b9bb5855d3566b86 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:28.926890 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25ce93e8577248a4b915265fb781faae" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:28.927197 4443 raft_consensus.cc:2379] T 25ce93e8577248a4b915265fb781faae P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 73a1f2ca1f544de9b0e3b37015f85328 in current term 1: Already voted for candidate 398c3c8f3195492c900439ad5e1931bb in this term.
I20240502 21:26:28.928328 4575 raft_consensus.cc:2454] T 23c09be2916844a9b9bb5855d3566b86 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:28.928601 4442 raft_consensus.cc:2454] T 23c09be2916844a9b9bb5855d3566b86 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:28.928865 4773 leader_election.cc:302] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:28.929119 4770 leader_election.cc:302] T 25ce93e8577248a4b915265fb781faae P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328; no voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1
I20240502 21:26:28.929316 5444 raft_consensus.cc:2781] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:28.929705 5460 raft_consensus.cc:2726] T 25ce93e8577248a4b915265fb781faae P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 21:26:28.930037 5444 raft_consensus.cc:686] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:26:28.930639 5444 consensus_queue.cc:237] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
W20240502 21:26:28.933161 4885 tablet.cc:2369] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:28.934098 4300 catalog_manager.cc:5484] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 398c3c8f3195492c900439ad5e1931bb (127.29.82.67). New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:26:28.935745 4299 catalog_manager.cc:5484] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
W20240502 21:26:28.965729 4753 tablet.cc:2369] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 21:26:28.992630 4622 tablet.cc:2369] T 23c09be2916844a9b9bb5855d3566b86 P 7a736a8ebb164ce0a051c0fccd76c3b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:29.005151 5454 raft_consensus.cc:482] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:29.005364 5454 raft_consensus.cc:504] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:29.007107 5454 leader_election.cc:288] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:29.014918 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7feb601648e14a998cfc7331a6934ac1" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:29.014918 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7feb601648e14a998cfc7331a6934ac1" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:29.015220 4442 raft_consensus.cc:2454] T 7feb601648e14a998cfc7331a6934ac1 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:26:29.015216 4706 raft_consensus.cc:2454] T 7feb601648e14a998cfc7331a6934ac1 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:26:29.015841 4511 leader_election.cc:302] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:29.016323 5454 raft_consensus.cc:2781] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:29.016460 5454 raft_consensus.cc:482] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:29.016573 5454 raft_consensus.cc:3037] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:29.018651 5454 raft_consensus.cc:504] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:29.019326 5454 leader_election.cc:288] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:29.019795 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7feb601648e14a998cfc7331a6934ac1" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:29.019901 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7feb601648e14a998cfc7331a6934ac1" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:29.020036 4442 raft_consensus.cc:3037] T 7feb601648e14a998cfc7331a6934ac1 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:29.020136 4706 raft_consensus.cc:3037] T 7feb601648e14a998cfc7331a6934ac1 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:29.021944 4442 raft_consensus.cc:2454] T 7feb601648e14a998cfc7331a6934ac1 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:26:29.022156 4706 raft_consensus.cc:2454] T 7feb601648e14a998cfc7331a6934ac1 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:26:29.022385 4508 leader_election.cc:302] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:29.022724 5454 raft_consensus.cc:2781] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:29.023144 5454 raft_consensus.cc:686] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:26:29.023737 5454 consensus_queue.cc:237] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:29.028048 4299 catalog_manager.cc:5484] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
W20240502 21:26:29.038815 4490 tablet.cc:2369] T 23c09be2916844a9b9bb5855d3566b86 P 71e1375b799b439fa3c83ca763c039c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:29.377414 5446 consensus_queue.cc:1035] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:29.387766 5446 consensus_queue.cc:1035] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:29.409375 5469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:29.409847 5469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:29.423134 5469 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:26:29.444638 5444 consensus_queue.cc:1035] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:29.451865 5444 consensus_queue.cc:1035] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240502 21:26:29.494961 5454 consensus_queue.cc:1035] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:29.506618 5467 consensus_queue.cc:1035] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 272.771 ms
time per row: 0.136386 ms
-----------------------------------------------------
Suppressions used:
count bytes template
28 7248 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:26:30.779743 4299 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:30.789404 4685 tablet_service.cc:1467] Processing CreateTablet for tablet eb81ce7407ab4c9e919a8a5f1b98d545 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6c511862aad4ba5a222ddedb7e95c27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:30.789734 4817 tablet_service.cc:1467] Processing CreateTablet for tablet eb81ce7407ab4c9e919a8a5f1b98d545 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6c511862aad4ba5a222ddedb7e95c27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:30.789983 4685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb81ce7407ab4c9e919a8a5f1b98d545. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:30.790235 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb81ce7407ab4c9e919a8a5f1b98d545. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:30.790235 4818 tablet_service.cc:1467] Processing CreateTablet for tablet 86fa4255c6554ba28e141de3cb328976 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6c511862aad4ba5a222ddedb7e95c27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:30.790565 4555 tablet_service.cc:1467] Processing CreateTablet for tablet eb81ce7407ab4c9e919a8a5f1b98d545 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6c511862aad4ba5a222ddedb7e95c27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:30.790755 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86fa4255c6554ba28e141de3cb328976. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:30.790992 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb81ce7407ab4c9e919a8a5f1b98d545. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:30.793830 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 86fa4255c6554ba28e141de3cb328976 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6c511862aad4ba5a222ddedb7e95c27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:30.793887 4684 tablet_service.cc:1467] Processing CreateTablet for tablet e528100a026a4b429ee146912d08a20e (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6c511862aad4ba5a222ddedb7e95c27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:30.794034 4421 tablet_service.cc:1467] Processing CreateTablet for tablet e528100a026a4b429ee146912d08a20e (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6c511862aad4ba5a222ddedb7e95c27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:30.794245 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86fa4255c6554ba28e141de3cb328976. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:30.794385 4554 tablet_service.cc:1467] Processing CreateTablet for tablet 86fa4255c6554ba28e141de3cb328976 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6c511862aad4ba5a222ddedb7e95c27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:30.794461 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e528100a026a4b429ee146912d08a20e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:30.794653 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86fa4255c6554ba28e141de3cb328976. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:30.799052 5537 tablet_bootstrap.cc:492] T 86fa4255c6554ba28e141de3cb328976 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:30.799096 4553 tablet_service.cc:1467] Processing CreateTablet for tablet e528100a026a4b429ee146912d08a20e (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6c511862aad4ba5a222ddedb7e95c27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:30.799465 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e528100a026a4b429ee146912d08a20e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:30.799818 4684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e528100a026a4b429ee146912d08a20e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:30.800863 5537 tablet_bootstrap.cc:654] T 86fa4255c6554ba28e141de3cb328976 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:30.808992 5540 tablet_bootstrap.cc:492] T e528100a026a4b429ee146912d08a20e P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:30.811236 5540 tablet_bootstrap.cc:654] T e528100a026a4b429ee146912d08a20e P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:30.813040 5539 tablet_bootstrap.cc:492] T eb81ce7407ab4c9e919a8a5f1b98d545 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:30.815660 5539 tablet_bootstrap.cc:654] T eb81ce7407ab4c9e919a8a5f1b98d545 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:30.817015 5537 tablet_bootstrap.cc:492] T 86fa4255c6554ba28e141de3cb328976 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:30.817238 5537 ts_tablet_manager.cc:1367] T 86fa4255c6554ba28e141de3cb328976 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.003s
I20240502 21:26:30.818248 5537 raft_consensus.cc:348] T 86fa4255c6554ba28e141de3cb328976 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:30.818686 5537 raft_consensus.cc:374] T 86fa4255c6554ba28e141de3cb328976 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:30.818857 5537 raft_consensus.cc:729] T 86fa4255c6554ba28e141de3cb328976 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:30.819248 5537 consensus_queue.cc:260] T 86fa4255c6554ba28e141de3cb328976 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:30.820246 5537 ts_tablet_manager.cc:1398] T 86fa4255c6554ba28e141de3cb328976 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:30.820678 5537 tablet_bootstrap.cc:492] T eb81ce7407ab4c9e919a8a5f1b98d545 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:30.822643 5537 tablet_bootstrap.cc:654] T eb81ce7407ab4c9e919a8a5f1b98d545 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:30.824146 5540 tablet_bootstrap.cc:492] T e528100a026a4b429ee146912d08a20e P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:30.824425 5540 ts_tablet_manager.cc:1367] T e528100a026a4b429ee146912d08a20e P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.009s
I20240502 21:26:30.826045 5540 raft_consensus.cc:348] T e528100a026a4b429ee146912d08a20e P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:30.826417 5540 raft_consensus.cc:374] T e528100a026a4b429ee146912d08a20e P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:30.826455 5541 tablet_bootstrap.cc:492] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:30.826598 5540 raft_consensus.cc:729] T e528100a026a4b429ee146912d08a20e P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:30.827014 5540 consensus_queue.cc:260] T e528100a026a4b429ee146912d08a20e P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:30.830725 5541 tablet_bootstrap.cc:654] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:30.831543 5539 tablet_bootstrap.cc:492] T eb81ce7407ab4c9e919a8a5f1b98d545 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:30.831874 5539 ts_tablet_manager.cc:1367] T eb81ce7407ab4c9e919a8a5f1b98d545 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.013s
I20240502 21:26:30.832681 5540 ts_tablet_manager.cc:1398] T e528100a026a4b429ee146912d08a20e P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.008s user 0.002s sys 0.004s
I20240502 21:26:30.832834 5539 raft_consensus.cc:348] T eb81ce7407ab4c9e919a8a5f1b98d545 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:30.833010 5539 raft_consensus.cc:374] T eb81ce7407ab4c9e919a8a5f1b98d545 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:30.833143 5539 raft_consensus.cc:729] T eb81ce7407ab4c9e919a8a5f1b98d545 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:30.833246 5537 tablet_bootstrap.cc:492] T eb81ce7407ab4c9e919a8a5f1b98d545 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:30.833371 5539 consensus_queue.cc:260] T eb81ce7407ab4c9e919a8a5f1b98d545 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:30.833513 5537 ts_tablet_manager.cc:1367] T eb81ce7407ab4c9e919a8a5f1b98d545 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20240502 21:26:30.834003 5539 ts_tablet_manager.cc:1398] T eb81ce7407ab4c9e919a8a5f1b98d545 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:26:30.834342 5539 tablet_bootstrap.cc:492] T 86fa4255c6554ba28e141de3cb328976 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:30.834522 5537 raft_consensus.cc:348] T eb81ce7407ab4c9e919a8a5f1b98d545 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:30.834753 5537 raft_consensus.cc:374] T eb81ce7407ab4c9e919a8a5f1b98d545 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:30.834990 5537 raft_consensus.cc:729] T eb81ce7407ab4c9e919a8a5f1b98d545 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:30.835348 5537 consensus_queue.cc:260] T eb81ce7407ab4c9e919a8a5f1b98d545 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:30.835983 5539 tablet_bootstrap.cc:654] T 86fa4255c6554ba28e141de3cb328976 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:30.836108 5537 ts_tablet_manager.cc:1398] T eb81ce7407ab4c9e919a8a5f1b98d545 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:30.836462 5537 tablet_bootstrap.cc:492] T e528100a026a4b429ee146912d08a20e P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:30.838228 5537 tablet_bootstrap.cc:654] T e528100a026a4b429ee146912d08a20e P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:30.839397 5541 tablet_bootstrap.cc:492] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:30.839623 5541 ts_tablet_manager.cc:1367] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20240502 21:26:30.840773 5541 raft_consensus.cc:348] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:30.841153 5541 raft_consensus.cc:374] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:30.841312 5541 raft_consensus.cc:729] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:30.841674 5541 consensus_queue.cc:260] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:30.842612 5541 ts_tablet_manager.cc:1398] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:30.843024 5541 tablet_bootstrap.cc:492] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:30.845415 5539 tablet_bootstrap.cc:492] T 86fa4255c6554ba28e141de3cb328976 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:30.845639 5539 ts_tablet_manager.cc:1367] T 86fa4255c6554ba28e141de3cb328976 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.001s
I20240502 21:26:30.846725 5537 tablet_bootstrap.cc:492] T e528100a026a4b429ee146912d08a20e P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:30.846908 5537 ts_tablet_manager.cc:1367] T e528100a026a4b429ee146912d08a20e P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240502 21:26:30.846464 5539 raft_consensus.cc:348] T 86fa4255c6554ba28e141de3cb328976 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:30.847077 5539 raft_consensus.cc:374] T 86fa4255c6554ba28e141de3cb328976 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:30.847157 5539 raft_consensus.cc:729] T 86fa4255c6554ba28e141de3cb328976 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:30.847368 5541 tablet_bootstrap.cc:654] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:30.847409 5539 consensus_queue.cc:260] T 86fa4255c6554ba28e141de3cb328976 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:30.847841 5537 raft_consensus.cc:348] T e528100a026a4b429ee146912d08a20e P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:30.847961 5539 ts_tablet_manager.cc:1398] T 86fa4255c6554ba28e141de3cb328976 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:30.848099 5537 raft_consensus.cc:374] T e528100a026a4b429ee146912d08a20e P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:30.848230 5539 tablet_bootstrap.cc:492] T e528100a026a4b429ee146912d08a20e P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:30.848299 5537 raft_consensus.cc:729] T e528100a026a4b429ee146912d08a20e P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:30.848657 5537 consensus_queue.cc:260] T e528100a026a4b429ee146912d08a20e P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:30.849541 5537 ts_tablet_manager.cc:1398] T e528100a026a4b429ee146912d08a20e P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:30.849845 5539 tablet_bootstrap.cc:654] T e528100a026a4b429ee146912d08a20e P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:30.850761 5541 tablet_bootstrap.cc:492] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:30.850971 5541 ts_tablet_manager.cc:1367] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20240502 21:26:30.852263 5539 tablet_bootstrap.cc:492] T e528100a026a4b429ee146912d08a20e P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:30.852409 5539 ts_tablet_manager.cc:1367] T e528100a026a4b429ee146912d08a20e P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 21:26:30.852875 5541 raft_consensus.cc:348] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:30.853057 5541 raft_consensus.cc:374] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:30.853003 5539 raft_consensus.cc:348] T e528100a026a4b429ee146912d08a20e P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:30.853134 5541 raft_consensus.cc:729] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:30.853178 5539 raft_consensus.cc:374] T e528100a026a4b429ee146912d08a20e P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:30.853268 5539 raft_consensus.cc:729] T e528100a026a4b429ee146912d08a20e P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:30.853379 5541 consensus_queue.cc:260] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:30.853473 5539 consensus_queue.cc:260] T e528100a026a4b429ee146912d08a20e P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:30.853850 5541 ts_tablet_manager.cc:1398] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240502 21:26:30.853979 5539 ts_tablet_manager.cc:1398] T e528100a026a4b429ee146912d08a20e P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 21:26:30.869002 5486 raft_consensus.cc:482] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:30.869179 5486 raft_consensus.cc:504] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:30.869786 5486 leader_election.cc:288] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:30.870330 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86fa4255c6554ba28e141de3cb328976" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:30.870330 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86fa4255c6554ba28e141de3cb328976" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:30.870558 4575 raft_consensus.cc:2454] T 86fa4255c6554ba28e141de3cb328976 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:30.870558 4705 raft_consensus.cc:2454] T 86fa4255c6554ba28e141de3cb328976 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:30.870992 4773 leader_election.cc:302] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:30.871243 5486 raft_consensus.cc:2781] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:30.871361 5486 raft_consensus.cc:482] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:30.871457 5486 raft_consensus.cc:3037] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:30.873308 5486 raft_consensus.cc:504] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:30.873829 5486 leader_election.cc:288] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:30.874310 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86fa4255c6554ba28e141de3cb328976" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:30.874310 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86fa4255c6554ba28e141de3cb328976" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:26:30.874502 4575 raft_consensus.cc:3037] T 86fa4255c6554ba28e141de3cb328976 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:30.874502 4705 raft_consensus.cc:3037] T 86fa4255c6554ba28e141de3cb328976 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:30.877346 4705 raft_consensus.cc:2454] T 86fa4255c6554ba28e141de3cb328976 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:30.877346 4575 raft_consensus.cc:2454] T 86fa4255c6554ba28e141de3cb328976 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:30.877753 4773 leader_election.cc:302] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:30.878023 5486 raft_consensus.cc:2781] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:30.878394 5486 raft_consensus.cc:686] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:26:30.878671 5486 consensus_queue.cc:237] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:30.881439 4299 catalog_manager.cc:5484] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:26:31.026942 5486 raft_consensus.cc:482] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:31.027173 5486 raft_consensus.cc:504] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:31.027956 5486 leader_election.cc:288] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:31.028502 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb81ce7407ab4c9e919a8a5f1b98d545" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:31.028800 4575 raft_consensus.cc:2454] T eb81ce7407ab4c9e919a8a5f1b98d545 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:31.028738 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb81ce7407ab4c9e919a8a5f1b98d545" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:31.029043 4705 raft_consensus.cc:2454] T eb81ce7407ab4c9e919a8a5f1b98d545 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:31.029259 4773 leader_election.cc:302] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:31.029608 5486 raft_consensus.cc:2781] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:31.029712 5486 raft_consensus.cc:482] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:31.029829 5486 raft_consensus.cc:3037] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:31.031778 5486 raft_consensus.cc:504] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:31.032313 5486 leader_election.cc:288] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:31.032692 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb81ce7407ab4c9e919a8a5f1b98d545" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:26:31.032805 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb81ce7407ab4c9e919a8a5f1b98d545" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:31.032884 4575 raft_consensus.cc:3037] T eb81ce7407ab4c9e919a8a5f1b98d545 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:31.033020 4705 raft_consensus.cc:3037] T eb81ce7407ab4c9e919a8a5f1b98d545 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:31.034843 4575 raft_consensus.cc:2454] T eb81ce7407ab4c9e919a8a5f1b98d545 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:31.035041 4705 raft_consensus.cc:2454] T eb81ce7407ab4c9e919a8a5f1b98d545 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:31.035234 4773 leader_election.cc:302] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:31.035507 5486 raft_consensus.cc:2781] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:31.035727 5486 raft_consensus.cc:686] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:26:31.035997 5486 consensus_queue.cc:237] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:31.038803 4299 catalog_manager.cc:5484] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:26:31.063539 5502 raft_consensus.cc:482] T e528100a026a4b429ee146912d08a20e P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:31.063781 5502 raft_consensus.cc:504] T e528100a026a4b429ee146912d08a20e P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:31.064429 5502 leader_election.cc:288] T e528100a026a4b429ee146912d08a20e P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:31.065006 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e528100a026a4b429ee146912d08a20e" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:31.065090 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e528100a026a4b429ee146912d08a20e" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:31.065263 4442 raft_consensus.cc:2454] T e528100a026a4b429ee146912d08a20e P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:26:31.065317 4705 raft_consensus.cc:2454] T e528100a026a4b429ee146912d08a20e P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:26:31.065749 4511 leader_election.cc:302] T e528100a026a4b429ee146912d08a20e P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:31.066009 5502 raft_consensus.cc:2781] T e528100a026a4b429ee146912d08a20e P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:31.066112 5502 raft_consensus.cc:482] T e528100a026a4b429ee146912d08a20e P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:31.066210 5502 raft_consensus.cc:3037] T e528100a026a4b429ee146912d08a20e P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:31.068220 5502 raft_consensus.cc:504] T e528100a026a4b429ee146912d08a20e P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:31.068763 5502 leader_election.cc:288] T e528100a026a4b429ee146912d08a20e P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:31.069247 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e528100a026a4b429ee146912d08a20e" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:31.069331 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e528100a026a4b429ee146912d08a20e" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:31.069453 4442 raft_consensus.cc:3037] T e528100a026a4b429ee146912d08a20e P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:31.069537 4705 raft_consensus.cc:3037] T e528100a026a4b429ee146912d08a20e P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:31.072360 4705 raft_consensus.cc:2454] T e528100a026a4b429ee146912d08a20e P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:26:31.072387 4442 raft_consensus.cc:2454] T e528100a026a4b429ee146912d08a20e P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:26:31.072778 4508 leader_election.cc:302] T e528100a026a4b429ee146912d08a20e P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:31.073056 5502 raft_consensus.cc:2781] T e528100a026a4b429ee146912d08a20e P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:31.073441 5502 raft_consensus.cc:686] T e528100a026a4b429ee146912d08a20e P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:26:31.073753 5502 consensus_queue.cc:237] T e528100a026a4b429ee146912d08a20e P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:31.076339 4299 catalog_manager.cc:5484] T e528100a026a4b429ee146912d08a20e P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
W20240502 21:26:31.427958 5556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:31.428295 5556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:31.435807 5556 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:26:31.438613 5486 consensus_queue.cc:1035] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:31.443315 5486 consensus_queue.cc:1035] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:31.458501 5486 consensus_queue.cc:1035] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:31.462878 5486 consensus_queue.cc:1035] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240502 21:26:31.596045 4442 raft_consensus.cc:1261] T e528100a026a4b429ee146912d08a20e P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:31.597090 5502 consensus_queue.cc:1035] T e528100a026a4b429ee146912d08a20e P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:31.599220 4705 raft_consensus.cc:1261] T e528100a026a4b429ee146912d08a20e P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:31.599907 5502 consensus_queue.cc:1035] T e528100a026a4b429ee146912d08a20e P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:31.650233 5607 mvcc.cc:205] Tried to move back new op lower bound from 7023350544769232896 to 7023350542640488448. Current Snapshot: MvccSnapshot[applied={T|T < 7023350544769232896}]
W20240502 21:26:31.708134 4511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.67:34537, user_credentials={real_user=slave}} blocked reactor thread for 55791.9us
I20240502 21:26:31.713797 5596 mvcc.cc:205] Tried to move back new op lower bound from 7023350544769232896 to 7023350542640488448. Current Snapshot: MvccSnapshot[applied={T|T < 7023350544769232896}]
I20240502 21:26:31.723806 5600 mvcc.cc:205] Tried to move back new op lower bound from 7023350544769232896 to 7023350542640488448. Current Snapshot: MvccSnapshot[applied={T|T < 7023350544769232896}]
INSERT report
rows total: 2000
time total: 254.721 ms
time per row: 0.12736 ms
I20240502 21:26:31.841534 4298 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:31.850476 4421 tablet_service.cc:1467] Processing CreateTablet for tablet eaef07076cf1438b8aac5ac1524bfccf (DEFAULT_TABLE table=rebalance_test_table_2 [id=d26a56879c874f3cb4fbd05df61cd275]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:31.850710 4553 tablet_service.cc:1467] Processing CreateTablet for tablet eaef07076cf1438b8aac5ac1524bfccf (DEFAULT_TABLE table=rebalance_test_table_2 [id=d26a56879c874f3cb4fbd05df61cd275]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:31.850996 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaef07076cf1438b8aac5ac1524bfccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:31.851197 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaef07076cf1438b8aac5ac1524bfccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:31.851114 4818 tablet_service.cc:1467] Processing CreateTablet for tablet eaef07076cf1438b8aac5ac1524bfccf (DEFAULT_TABLE table=rebalance_test_table_2 [id=d26a56879c874f3cb4fbd05df61cd275]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:31.851548 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaef07076cf1438b8aac5ac1524bfccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:31.856035 5624 tablet_bootstrap.cc:492] T eaef07076cf1438b8aac5ac1524bfccf P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:31.857777 4817 tablet_service.cc:1467] Processing CreateTablet for tablet e0768388c4c542aab05a42d5ebc78f93 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d26a56879c874f3cb4fbd05df61cd275]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:31.857996 5624 tablet_bootstrap.cc:654] T eaef07076cf1438b8aac5ac1524bfccf P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:31.858165 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0768388c4c542aab05a42d5ebc78f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:31.859300 4685 tablet_service.cc:1467] Processing CreateTablet for tablet e0768388c4c542aab05a42d5ebc78f93 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d26a56879c874f3cb4fbd05df61cd275]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:31.859874 4685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0768388c4c542aab05a42d5ebc78f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:31.860164 4423 tablet_service.cc:1467] Processing CreateTablet for tablet e0768388c4c542aab05a42d5ebc78f93 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d26a56879c874f3cb4fbd05df61cd275]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:31.860685 4423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0768388c4c542aab05a42d5ebc78f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:31.864109 4422 tablet_service.cc:1467] Processing CreateTablet for tablet f134f962f5024e128db94eea8527712f (DEFAULT_TABLE table=rebalance_test_table_2 [id=d26a56879c874f3cb4fbd05df61cd275]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:31.864645 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f134f962f5024e128db94eea8527712f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:31.865622 4553 tablet_service.cc:1467] Processing CreateTablet for tablet f134f962f5024e128db94eea8527712f (DEFAULT_TABLE table=rebalance_test_table_2 [id=d26a56879c874f3cb4fbd05df61cd275]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:31.866020 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f134f962f5024e128db94eea8527712f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:31.874289 5624 tablet_bootstrap.cc:492] T eaef07076cf1438b8aac5ac1524bfccf P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:31.874543 5624 ts_tablet_manager.cc:1367] T eaef07076cf1438b8aac5ac1524bfccf P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.007s
I20240502 21:26:31.875504 5627 tablet_bootstrap.cc:492] T e0768388c4c542aab05a42d5ebc78f93 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:31.875624 5624 raft_consensus.cc:348] T eaef07076cf1438b8aac5ac1524bfccf P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:31.875839 5624 raft_consensus.cc:374] T eaef07076cf1438b8aac5ac1524bfccf P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:31.875929 5624 raft_consensus.cc:729] T eaef07076cf1438b8aac5ac1524bfccf P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:31.876222 5624 consensus_queue.cc:260] T eaef07076cf1438b8aac5ac1524bfccf P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:31.881626 5624 ts_tablet_manager.cc:1398] T eaef07076cf1438b8aac5ac1524bfccf P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 21:26:31.881950 5624 tablet_bootstrap.cc:492] T f134f962f5024e128db94eea8527712f P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:31.883311 4816 tablet_service.cc:1467] Processing CreateTablet for tablet f134f962f5024e128db94eea8527712f (DEFAULT_TABLE table=rebalance_test_table_2 [id=d26a56879c874f3cb4fbd05df61cd275]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:31.883847 4816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f134f962f5024e128db94eea8527712f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:31.885248 5624 tablet_bootstrap.cc:654] T f134f962f5024e128db94eea8527712f P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:31.894845 5625 tablet_bootstrap.cc:492] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:31.895411 5628 tablet_bootstrap.cc:492] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:31.896275 5624 tablet_bootstrap.cc:492] T f134f962f5024e128db94eea8527712f P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:31.896494 5624 ts_tablet_manager.cc:1367] T f134f962f5024e128db94eea8527712f P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.010s
I20240502 21:26:31.897171 5627 tablet_bootstrap.cc:654] T e0768388c4c542aab05a42d5ebc78f93 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:31.897532 5625 tablet_bootstrap.cc:654] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:31.897392 5624 raft_consensus.cc:348] T f134f962f5024e128db94eea8527712f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:31.897599 5628 tablet_bootstrap.cc:654] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:31.897634 5624 raft_consensus.cc:374] T f134f962f5024e128db94eea8527712f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:31.897704 5624 raft_consensus.cc:729] T f134f962f5024e128db94eea8527712f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:31.898026 5624 consensus_queue.cc:260] T f134f962f5024e128db94eea8527712f P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:31.898684 5624 ts_tablet_manager.cc:1398] T f134f962f5024e128db94eea8527712f P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:31.901343 5625 tablet_bootstrap.cc:492] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:31.901491 5625 ts_tablet_manager.cc:1367] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 21:26:31.902289 5625 raft_consensus.cc:348] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:31.902580 5625 raft_consensus.cc:374] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:31.902742 5625 raft_consensus.cc:729] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:31.903062 5625 consensus_queue.cc:260] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:31.904371 5625 ts_tablet_manager.cc:1398] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:31.904716 5625 tablet_bootstrap.cc:492] T e0768388c4c542aab05a42d5ebc78f93 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:31.906155 5628 tablet_bootstrap.cc:492] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:31.906392 5628 ts_tablet_manager.cc:1367] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240502 21:26:31.906826 5625 tablet_bootstrap.cc:654] T e0768388c4c542aab05a42d5ebc78f93 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:31.907323 5628 raft_consensus.cc:348] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:31.907615 5627 tablet_bootstrap.cc:492] T e0768388c4c542aab05a42d5ebc78f93 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:31.907625 5628 raft_consensus.cc:374] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:31.907788 5628 raft_consensus.cc:729] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:31.907833 5627 ts_tablet_manager.cc:1367] T e0768388c4c542aab05a42d5ebc78f93 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.008s
I20240502 21:26:31.908113 5628 consensus_queue.cc:260] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:31.908908 5628 ts_tablet_manager.cc:1398] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:31.909229 5628 tablet_bootstrap.cc:492] T eaef07076cf1438b8aac5ac1524bfccf P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:31.909538 5627 raft_consensus.cc:348] T e0768388c4c542aab05a42d5ebc78f93 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:31.909804 5627 raft_consensus.cc:374] T e0768388c4c542aab05a42d5ebc78f93 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:31.909941 5627 raft_consensus.cc:729] T e0768388c4c542aab05a42d5ebc78f93 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:31.910296 5627 consensus_queue.cc:260] T e0768388c4c542aab05a42d5ebc78f93 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:31.911499 5628 tablet_bootstrap.cc:654] T eaef07076cf1438b8aac5ac1524bfccf P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:31.914218 5627 ts_tablet_manager.cc:1398] T e0768388c4c542aab05a42d5ebc78f93 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 21:26:31.915521 5625 tablet_bootstrap.cc:492] T e0768388c4c542aab05a42d5ebc78f93 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:31.915784 5625 ts_tablet_manager.cc:1367] T e0768388c4c542aab05a42d5ebc78f93 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.006s
I20240502 21:26:31.916334 5628 tablet_bootstrap.cc:492] T eaef07076cf1438b8aac5ac1524bfccf P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:31.916535 5628 ts_tablet_manager.cc:1367] T eaef07076cf1438b8aac5ac1524bfccf P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20240502 21:26:31.916563 5625 raft_consensus.cc:348] T e0768388c4c542aab05a42d5ebc78f93 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:31.916800 5625 raft_consensus.cc:374] T e0768388c4c542aab05a42d5ebc78f93 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:31.916882 5625 raft_consensus.cc:729] T e0768388c4c542aab05a42d5ebc78f93 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:31.917100 5625 consensus_queue.cc:260] T e0768388c4c542aab05a42d5ebc78f93 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:31.917233 5628 raft_consensus.cc:348] T eaef07076cf1438b8aac5ac1524bfccf P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:31.917466 5628 raft_consensus.cc:374] T eaef07076cf1438b8aac5ac1524bfccf P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:31.917543 5625 ts_tablet_manager.cc:1398] T e0768388c4c542aab05a42d5ebc78f93 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:26:31.917562 5628 raft_consensus.cc:729] T eaef07076cf1438b8aac5ac1524bfccf P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:31.917851 5625 tablet_bootstrap.cc:492] T f134f962f5024e128db94eea8527712f P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:31.917814 5628 consensus_queue.cc:260] T eaef07076cf1438b8aac5ac1524bfccf P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:31.918736 5628 ts_tablet_manager.cc:1398] T eaef07076cf1438b8aac5ac1524bfccf P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:31.919049 5628 tablet_bootstrap.cc:492] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:31.919787 5625 tablet_bootstrap.cc:654] T f134f962f5024e128db94eea8527712f P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:31.920635 5628 tablet_bootstrap.cc:654] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:31.925063 5628 tablet_bootstrap.cc:492] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:31.925063 5625 tablet_bootstrap.cc:492] T f134f962f5024e128db94eea8527712f P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:31.925210 5625 ts_tablet_manager.cc:1367] T f134f962f5024e128db94eea8527712f P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20240502 21:26:31.925210 5628 ts_tablet_manager.cc:1367] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240502 21:26:31.925936 5628 raft_consensus.cc:348] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:31.925951 5625 raft_consensus.cc:348] T f134f962f5024e128db94eea8527712f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:31.926139 5625 raft_consensus.cc:374] T f134f962f5024e128db94eea8527712f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:31.926149 5628 raft_consensus.cc:374] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:31.926246 5625 raft_consensus.cc:729] T f134f962f5024e128db94eea8527712f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:31.926247 5628 raft_consensus.cc:729] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:31.926467 5628 consensus_queue.cc:260] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:31.926465 5625 consensus_queue.cc:260] T f134f962f5024e128db94eea8527712f P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:31.926977 5628 ts_tablet_manager.cc:1398] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:26:31.926977 5625 ts_tablet_manager.cc:1398] T f134f962f5024e128db94eea8527712f P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:31.994005 5618 raft_consensus.cc:482] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:31.994227 5618 raft_consensus.cc:504] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:31.995193 5618 leader_election.cc:288] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:31.995623 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0768388c4c542aab05a42d5ebc78f93" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:31.995604 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0768388c4c542aab05a42d5ebc78f93" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:31.995908 4706 raft_consensus.cc:2454] T e0768388c4c542aab05a42d5ebc78f93 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:31.995887 4442 raft_consensus.cc:2454] T e0768388c4c542aab05a42d5ebc78f93 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:31.996389 4770 leader_election.cc:302] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:31.996655 5618 raft_consensus.cc:2781] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:31.996768 5618 raft_consensus.cc:482] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:31.996871 5618 raft_consensus.cc:3037] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:31.998916 5618 raft_consensus.cc:504] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:31.999505 5618 leader_election.cc:288] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:31.999928 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0768388c4c542aab05a42d5ebc78f93" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:31.999923 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0768388c4c542aab05a42d5ebc78f93" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:32.000102 4442 raft_consensus.cc:3037] T e0768388c4c542aab05a42d5ebc78f93 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:32.000135 4706 raft_consensus.cc:3037] T e0768388c4c542aab05a42d5ebc78f93 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:32.001889 4442 raft_consensus.cc:2454] T e0768388c4c542aab05a42d5ebc78f93 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:32.002286 4770 leader_election.cc:302] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:32.002579 5618 raft_consensus.cc:2781] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:32.002724 5618 raft_consensus.cc:686] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:26:32.002930 4706 raft_consensus.cc:2454] T e0768388c4c542aab05a42d5ebc78f93 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:32.003046 5618 consensus_queue.cc:237] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:32.006358 4298 catalog_manager.cc:5484] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:26:32.045800 5604 raft_consensus.cc:482] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:32.046022 5604 raft_consensus.cc:504] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:32.047647 5604 leader_election.cc:288] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:32.054248 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaef07076cf1438b8aac5ac1524bfccf" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:32.054561 4575 raft_consensus.cc:2454] T eaef07076cf1438b8aac5ac1524bfccf P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:26:32.054721 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaef07076cf1438b8aac5ac1524bfccf" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:26:32.054961 4839 raft_consensus.cc:2454] T eaef07076cf1438b8aac5ac1524bfccf P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:26:32.055111 4378 leader_election.cc:302] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:32.055554 5604 raft_consensus.cc:2781] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:32.055707 5604 raft_consensus.cc:482] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:32.055812 5604 raft_consensus.cc:3037] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:32.057726 5604 raft_consensus.cc:504] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:32.058343 5604 leader_election.cc:288] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:32.058765 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaef07076cf1438b8aac5ac1524bfccf" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:26:32.058848 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaef07076cf1438b8aac5ac1524bfccf" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:26:32.058990 4575 raft_consensus.cc:3037] T eaef07076cf1438b8aac5ac1524bfccf P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:32.059057 4839 raft_consensus.cc:3037] T eaef07076cf1438b8aac5ac1524bfccf P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:32.061005 4575 raft_consensus.cc:2454] T eaef07076cf1438b8aac5ac1524bfccf P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:26:32.061013 4839 raft_consensus.cc:2454] T eaef07076cf1438b8aac5ac1524bfccf P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:26:32.061367 4378 leader_election.cc:302] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:32.061676 5604 raft_consensus.cc:2781] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:32.062062 5604 raft_consensus.cc:686] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:26:32.062472 5604 consensus_queue.cc:237] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:32.065691 4298 catalog_manager.cc:5484] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:26:32.091662 5618 raft_consensus.cc:482] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:32.091885 5618 raft_consensus.cc:504] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:32.092522 5618 leader_election.cc:288] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:32.093179 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f134f962f5024e128db94eea8527712f" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:32.093178 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f134f962f5024e128db94eea8527712f" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:32.093457 4442 raft_consensus.cc:2454] T f134f962f5024e128db94eea8527712f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:32.093457 4575 raft_consensus.cc:2454] T f134f962f5024e128db94eea8527712f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:32.093909 4773 leader_election.cc:302] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:32.094206 5618 raft_consensus.cc:2781] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:32.094317 5618 raft_consensus.cc:482] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:32.094434 5618 raft_consensus.cc:3037] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:32.096518 5618 raft_consensus.cc:504] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:32.097085 5618 leader_election.cc:288] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:32.097568 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f134f962f5024e128db94eea8527712f" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:26:32.097571 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f134f962f5024e128db94eea8527712f" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:32.097767 4442 raft_consensus.cc:3037] T f134f962f5024e128db94eea8527712f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:32.097792 4575 raft_consensus.cc:3037] T f134f962f5024e128db94eea8527712f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:32.099671 4442 raft_consensus.cc:2454] T f134f962f5024e128db94eea8527712f P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:32.099675 4575 raft_consensus.cc:2454] T f134f962f5024e128db94eea8527712f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:32.100085 4773 leader_election.cc:302] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:32.100342 5618 raft_consensus.cc:2781] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:32.100514 5618 raft_consensus.cc:686] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:26:32.100775 5618 consensus_queue.cc:237] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:32.103363 4298 catalog_manager.cc:5484] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:26:32.410006 5618 consensus_queue.cc:1035] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:32.416065 5486 consensus_queue.cc:1035] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:32.485671 5486 consensus_queue.cc:1035] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:32.490789 5618 consensus_queue.cc:1035] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:32.532970 5643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:32.533285 5643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:32.541126 5643 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 21:26:32.601480 5604 consensus_queue.cc:1035] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:32.633540 5604 consensus_queue.cc:1035] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
INSERT report
rows total: 2000
time total: 275.401 ms
time per row: 0.1377 ms
I20240502 21:26:32.959962 4295 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:32.969375 4685 tablet_service.cc:1467] Processing CreateTablet for tablet 14bd28b4c6a54d98a18351b60f299a3a (DEFAULT_TABLE table=rebalance_test_table_3 [id=3c51dfaf2d824f93813d8d39758c9a8c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:32.969574 4553 tablet_service.cc:1467] Processing CreateTablet for tablet 14bd28b4c6a54d98a18351b60f299a3a (DEFAULT_TABLE table=rebalance_test_table_3 [id=3c51dfaf2d824f93813d8d39758c9a8c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:32.969941 4685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14bd28b4c6a54d98a18351b60f299a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:32.970028 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14bd28b4c6a54d98a18351b60f299a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:32.969987 4421 tablet_service.cc:1467] Processing CreateTablet for tablet 14bd28b4c6a54d98a18351b60f299a3a (DEFAULT_TABLE table=rebalance_test_table_3 [id=3c51dfaf2d824f93813d8d39758c9a8c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:32.970438 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14bd28b4c6a54d98a18351b60f299a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:32.971596 4684 tablet_service.cc:1467] Processing CreateTablet for tablet b5099940a48f4466aa7c713bb88d108e (DEFAULT_TABLE table=rebalance_test_table_3 [id=3c51dfaf2d824f93813d8d39758c9a8c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:32.972002 4684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5099940a48f4466aa7c713bb88d108e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:32.974174 4554 tablet_service.cc:1467] Processing CreateTablet for tablet 016ad8b6874c4b629882f1243ea1fba2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3c51dfaf2d824f93813d8d39758c9a8c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:32.974579 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 016ad8b6874c4b629882f1243ea1fba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:32.976836 4817 tablet_service.cc:1467] Processing CreateTablet for tablet b5099940a48f4466aa7c713bb88d108e (DEFAULT_TABLE table=rebalance_test_table_3 [id=3c51dfaf2d824f93813d8d39758c9a8c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:32.976863 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 016ad8b6874c4b629882f1243ea1fba2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3c51dfaf2d824f93813d8d39758c9a8c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:32.977231 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 016ad8b6874c4b629882f1243ea1fba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:32.977274 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5099940a48f4466aa7c713bb88d108e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:32.979935 4816 tablet_service.cc:1467] Processing CreateTablet for tablet 016ad8b6874c4b629882f1243ea1fba2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3c51dfaf2d824f93813d8d39758c9a8c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:32.980304 4816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 016ad8b6874c4b629882f1243ea1fba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:32.986804 4423 tablet_service.cc:1467] Processing CreateTablet for tablet b5099940a48f4466aa7c713bb88d108e (DEFAULT_TABLE table=rebalance_test_table_3 [id=3c51dfaf2d824f93813d8d39758c9a8c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:32.987368 4423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5099940a48f4466aa7c713bb88d108e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:33.006361 5706 tablet_bootstrap.cc:492] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:33.008435 5707 tablet_bootstrap.cc:492] T b5099940a48f4466aa7c713bb88d108e P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:33.008628 5706 tablet_bootstrap.cc:654] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:33.009919 5704 tablet_bootstrap.cc:492] T 14bd28b4c6a54d98a18351b60f299a3a P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:33.010391 5705 tablet_bootstrap.cc:492] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:33.012208 5707 tablet_bootstrap.cc:654] T b5099940a48f4466aa7c713bb88d108e P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:33.012220 5705 tablet_bootstrap.cc:654] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:33.013244 5704 tablet_bootstrap.cc:654] T 14bd28b4c6a54d98a18351b60f299a3a P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:33.021824 5704 tablet_bootstrap.cc:492] T 14bd28b4c6a54d98a18351b60f299a3a P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:33.022132 5704 ts_tablet_manager.cc:1367] T 14bd28b4c6a54d98a18351b60f299a3a P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20240502 21:26:33.023298 5704 raft_consensus.cc:348] T 14bd28b4c6a54d98a18351b60f299a3a P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:33.023798 5704 raft_consensus.cc:374] T 14bd28b4c6a54d98a18351b60f299a3a P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:33.023970 5704 raft_consensus.cc:729] T 14bd28b4c6a54d98a18351b60f299a3a P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:33.024446 5704 consensus_queue.cc:260] T 14bd28b4c6a54d98a18351b60f299a3a P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:33.025497 5704 ts_tablet_manager.cc:1398] T 14bd28b4c6a54d98a18351b60f299a3a P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:26:33.025926 5704 tablet_bootstrap.cc:492] T b5099940a48f4466aa7c713bb88d108e P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:33.028262 5704 tablet_bootstrap.cc:654] T b5099940a48f4466aa7c713bb88d108e P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:33.033797 5706 tablet_bootstrap.cc:492] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:33.034125 5706 ts_tablet_manager.cc:1367] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.008s
I20240502 21:26:33.035461 5706 raft_consensus.cc:348] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:33.035858 5706 raft_consensus.cc:374] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:33.036031 5706 raft_consensus.cc:729] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:33.036455 5706 consensus_queue.cc:260] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:33.037395 5706 ts_tablet_manager.cc:1398] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:33.037739 5706 tablet_bootstrap.cc:492] T 016ad8b6874c4b629882f1243ea1fba2 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:33.040436 5704 tablet_bootstrap.cc:492] T b5099940a48f4466aa7c713bb88d108e P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:33.040725 5704 ts_tablet_manager.cc:1367] T b5099940a48f4466aa7c713bb88d108e P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.007s
I20240502 21:26:33.041814 5704 raft_consensus.cc:348] T b5099940a48f4466aa7c713bb88d108e P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:33.042196 5704 raft_consensus.cc:374] T b5099940a48f4466aa7c713bb88d108e P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:33.042342 5704 raft_consensus.cc:729] T b5099940a48f4466aa7c713bb88d108e P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:33.043752 5704 consensus_queue.cc:260] T b5099940a48f4466aa7c713bb88d108e P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:33.044348 5705 tablet_bootstrap.cc:492] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:33.044925 5704 ts_tablet_manager.cc:1398] T b5099940a48f4466aa7c713bb88d108e P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 21:26:33.045153 5705 ts_tablet_manager.cc:1367] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.035s user 0.012s sys 0.020s
I20240502 21:26:33.045471 5704 tablet_bootstrap.cc:492] T 016ad8b6874c4b629882f1243ea1fba2 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:33.046303 5705 raft_consensus.cc:348] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:33.048102 5704 tablet_bootstrap.cc:654] T 016ad8b6874c4b629882f1243ea1fba2 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:33.048226 5705 raft_consensus.cc:374] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:33.048358 5705 raft_consensus.cc:729] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:33.048640 5705 consensus_queue.cc:260] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:33.049309 5705 ts_tablet_manager.cc:1398] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:26:33.049371 5707 tablet_bootstrap.cc:492] T b5099940a48f4466aa7c713bb88d108e P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:33.050401 5705 tablet_bootstrap.cc:492] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:33.051478 5707 ts_tablet_manager.cc:1367] T b5099940a48f4466aa7c713bb88d108e P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.043s user 0.014s sys 0.015s
I20240502 21:26:33.051257 5706 tablet_bootstrap.cc:654] T 016ad8b6874c4b629882f1243ea1fba2 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:33.052012 5705 tablet_bootstrap.cc:654] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:33.053603 5618 raft_consensus.cc:482] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:33.053786 5618 raft_consensus.cc:504] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:33.054081 5704 tablet_bootstrap.cc:492] T 016ad8b6874c4b629882f1243ea1fba2 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:33.054318 5704 ts_tablet_manager.cc:1367] T 016ad8b6874c4b629882f1243ea1fba2 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240502 21:26:33.054628 5618 leader_election.cc:288] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:33.055199 5704 raft_consensus.cc:348] T 016ad8b6874c4b629882f1243ea1fba2 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:33.055423 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5099940a48f4466aa7c713bb88d108e" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:33.055701 4706 raft_consensus.cc:2454] T b5099940a48f4466aa7c713bb88d108e P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:33.056541 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5099940a48f4466aa7c713bb88d108e" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:33.056627 4773 leader_election.cc:302] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:33.056949 5618 raft_consensus.cc:2781] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:33.057087 5618 raft_consensus.cc:482] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:33.057199 5618 raft_consensus.cc:3037] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:33.059474 5706 tablet_bootstrap.cc:492] T 016ad8b6874c4b629882f1243ea1fba2 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:33.059764 5706 ts_tablet_manager.cc:1367] T 016ad8b6874c4b629882f1243ea1fba2 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.001s
I20240502 21:26:33.060627 5618 raft_consensus.cc:504] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:33.060731 5706 raft_consensus.cc:348] T 016ad8b6874c4b629882f1243ea1fba2 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:33.061375 5706 raft_consensus.cc:374] T 016ad8b6874c4b629882f1243ea1fba2 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:33.061533 5706 raft_consensus.cc:729] T 016ad8b6874c4b629882f1243ea1fba2 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:33.061874 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5099940a48f4466aa7c713bb88d108e" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:33.061924 5706 consensus_queue.cc:260] T 016ad8b6874c4b629882f1243ea1fba2 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:33.062129 4706 raft_consensus.cc:3037] T b5099940a48f4466aa7c713bb88d108e P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:33.063380 5706 ts_tablet_manager.cc:1398] T 016ad8b6874c4b629882f1243ea1fba2 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:33.064548 5618 leader_election.cc:288] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:33.055421 5704 raft_consensus.cc:374] T 016ad8b6874c4b629882f1243ea1fba2 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:33.065021 5704 raft_consensus.cc:729] T 016ad8b6874c4b629882f1243ea1fba2 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:33.065043 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5099940a48f4466aa7c713bb88d108e" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:33.065366 5705 tablet_bootstrap.cc:492] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:33.065330 5704 consensus_queue.cc:260] T 016ad8b6874c4b629882f1243ea1fba2 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:33.065713 5705 ts_tablet_manager.cc:1367] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.003s
I20240502 21:26:33.066454 5705 raft_consensus.cc:348] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:33.067132 5707 raft_consensus.cc:348] T b5099940a48f4466aa7c713bb88d108e P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:33.067304 5705 raft_consensus.cc:374] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:33.067381 5707 raft_consensus.cc:374] T b5099940a48f4466aa7c713bb88d108e P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:33.067457 5705 raft_consensus.cc:729] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:33.067498 5707 raft_consensus.cc:729] T b5099940a48f4466aa7c713bb88d108e P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:33.067781 5707 consensus_queue.cc:260] T b5099940a48f4466aa7c713bb88d108e P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:33.068089 4443 raft_consensus.cc:2454] T b5099940a48f4466aa7c713bb88d108e P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:33.067807 5705 consensus_queue.cc:260] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:33.068439 5707 ts_tablet_manager.cc:1398] T b5099940a48f4466aa7c713bb88d108e P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240502 21:26:33.068511 5704 ts_tablet_manager.cc:1398] T 016ad8b6874c4b629882f1243ea1fba2 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240502 21:26:33.068847 5705 ts_tablet_manager.cc:1398] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:33.069290 5707 tablet_bootstrap.cc:492] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:33.070981 5707 tablet_bootstrap.cc:654] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:33.071017 4706 raft_consensus.cc:2454] T b5099940a48f4466aa7c713bb88d108e P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:33.071517 4773 leader_election.cc:302] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 73a1f2ca1f544de9b0e3b37015f85328; no voters: 71e1375b799b439fa3c83ca763c039c1
I20240502 21:26:33.072310 5618 raft_consensus.cc:2781] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:33.076068 5618 raft_consensus.cc:686] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:26:33.076424 5618 consensus_queue.cc:237] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:33.076840 5707 tablet_bootstrap.cc:492] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:33.077004 5707 ts_tablet_manager.cc:1367] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240502 21:26:33.077746 5707 raft_consensus.cc:348] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:33.077984 5707 raft_consensus.cc:374] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:33.078090 5707 raft_consensus.cc:729] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:33.078598 5707 consensus_queue.cc:260] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:33.079049 5707 ts_tablet_manager.cc:1398] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:33.078945 4296 catalog_manager.cc:5484] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:26:33.170289 5602 raft_consensus.cc:482] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:33.170495 5602 raft_consensus.cc:504] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:33.171277 5602 leader_election.cc:288] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:33.171926 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "016ad8b6874c4b629882f1243ea1fba2" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:33.172214 4706 raft_consensus.cc:2454] T 016ad8b6874c4b629882f1243ea1fba2 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:26:33.172626 4511 leader_election.cc:302] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:33.172998 5602 raft_consensus.cc:2781] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:33.173144 5602 raft_consensus.cc:482] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:33.173249 5602 raft_consensus.cc:3037] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:33.175733 5602 raft_consensus.cc:504] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:33.176576 5602 leader_election.cc:288] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:33.176817 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "016ad8b6874c4b629882f1243ea1fba2" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:33.177052 4706 raft_consensus.cc:3037] T 016ad8b6874c4b629882f1243ea1fba2 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:33.178565 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "016ad8b6874c4b629882f1243ea1fba2" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:26:33.178818 4839 raft_consensus.cc:2454] T 016ad8b6874c4b629882f1243ea1fba2 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:26:33.178794 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "016ad8b6874c4b629882f1243ea1fba2" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:26:33.179127 4838 raft_consensus.cc:3037] T 016ad8b6874c4b629882f1243ea1fba2 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:33.179741 4706 raft_consensus.cc:2454] T 016ad8b6874c4b629882f1243ea1fba2 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:26:33.180186 4511 leader_election.cc:302] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:33.180456 5602 raft_consensus.cc:2781] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:33.180866 5602 raft_consensus.cc:686] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:26:33.181174 5602 consensus_queue.cc:237] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:33.181558 4838 raft_consensus.cc:2454] T 016ad8b6874c4b629882f1243ea1fba2 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:26:33.183965 4294 catalog_manager.cc:5484] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:26:33.364655 5604 raft_consensus.cc:482] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:33.364864 5604 raft_consensus.cc:504] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:33.365538 5604 leader_election.cc:288] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:33.366648 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bd28b4c6a54d98a18351b60f299a3a" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:33.366889 4574 raft_consensus.cc:2454] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:26:33.367363 4378 leader_election.cc:302] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:33.367724 5604 raft_consensus.cc:2781] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:33.367817 5604 raft_consensus.cc:482] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:33.367923 5604 raft_consensus.cc:3037] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:33.370361 5461 raft_consensus.cc:482] T 14bd28b4c6a54d98a18351b60f299a3a P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:33.370517 5461 raft_consensus.cc:504] T 14bd28b4c6a54d98a18351b60f299a3a P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:33.370548 5604 raft_consensus.cc:504] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:33.371193 5604 leader_election.cc:288] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:33.371292 5461 leader_election.cc:288] T 14bd28b4c6a54d98a18351b60f299a3a P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:33.371969 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bd28b4c6a54d98a18351b60f299a3a" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:26:33.371991 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bd28b4c6a54d98a18351b60f299a3a" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:33.372162 4574 raft_consensus.cc:3037] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:33.372208 4443 raft_consensus.cc:2379] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 398c3c8f3195492c900439ad5e1931bb in current term 1: Already voted for candidate 71e1375b799b439fa3c83ca763c039c1 in this term.
I20240502 21:26:33.372598 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bd28b4c6a54d98a18351b60f299a3a" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:33.373282 4642 leader_election.cc:302] T 14bd28b4c6a54d98a18351b60f299a3a P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb; no voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:26:33.373576 5461 raft_consensus.cc:3037] T 14bd28b4c6a54d98a18351b60f299a3a P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:33.374186 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bd28b4c6a54d98a18351b60f299a3a" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:33.374558 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bd28b4c6a54d98a18351b60f299a3a" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:33.374877 4574 raft_consensus.cc:2454] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:26:33.375334 4378 leader_election.cc:302] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:33.375603 5604 raft_consensus.cc:2781] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:33.376051 5604 raft_consensus.cc:686] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:26:33.376335 5461 raft_consensus.cc:2726] T 14bd28b4c6a54d98a18351b60f299a3a P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:26:33.376334 5604 consensus_queue.cc:237] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:33.378638 4705 raft_consensus.cc:2454] T 14bd28b4c6a54d98a18351b60f299a3a P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:26:33.379154 4294 catalog_manager.cc:5484] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:26:33.578068 5602 consensus_queue.cc:1035] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:33.584473 5602 consensus_queue.cc:1035] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:33.606031 4443 raft_consensus.cc:3037] T b5099940a48f4466aa7c713bb88d108e P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:33.610227 5618 consensus_queue.cc:1035] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:33.616166 5618 consensus_queue.cc:1035] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:33.803963 5725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:33.804317 5725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:33.812444 5725 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:26:33.814527 5604 consensus_queue.cc:1035] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:33.819625 5723 consensus_queue.cc:1035] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 230.54 ms
time per row: 0.11527 ms
I20240502 21:26:34.195415 4293 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:34.205173 4553 tablet_service.cc:1467] Processing CreateTablet for tablet 20c21d242c8c4e00a45ae5de7388db3d (DEFAULT_TABLE table=rebalance_test_table_4 [id=2f224bc51e0b4008a30114b6f180f43b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:34.205451 4423 tablet_service.cc:1467] Processing CreateTablet for tablet 20c21d242c8c4e00a45ae5de7388db3d (DEFAULT_TABLE table=rebalance_test_table_4 [id=2f224bc51e0b4008a30114b6f180f43b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:34.205677 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c21d242c8c4e00a45ae5de7388db3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:34.205916 4423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c21d242c8c4e00a45ae5de7388db3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:34.206327 4421 tablet_service.cc:1467] Processing CreateTablet for tablet 18039778dfde4bcaa1245544e3d3c1a3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2f224bc51e0b4008a30114b6f180f43b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:34.206439 4685 tablet_service.cc:1467] Processing CreateTablet for tablet 20c21d242c8c4e00a45ae5de7388db3d (DEFAULT_TABLE table=rebalance_test_table_4 [id=2f224bc51e0b4008a30114b6f180f43b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:34.206691 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18039778dfde4bcaa1245544e3d3c1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:34.206905 4685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c21d242c8c4e00a45ae5de7388db3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:34.209200 4817 tablet_service.cc:1467] Processing CreateTablet for tablet 18039778dfde4bcaa1245544e3d3c1a3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2f224bc51e0b4008a30114b6f180f43b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:34.209633 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18039778dfde4bcaa1245544e3d3c1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:34.213064 4686 tablet_service.cc:1467] Processing CreateTablet for tablet f06f4eacdcec4b86a1bfd8e79be58bfe (DEFAULT_TABLE table=rebalance_test_table_4 [id=2f224bc51e0b4008a30114b6f180f43b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:34.213492 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f06f4eacdcec4b86a1bfd8e79be58bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:34.216203 4554 tablet_service.cc:1467] Processing CreateTablet for tablet 18039778dfde4bcaa1245544e3d3c1a3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2f224bc51e0b4008a30114b6f180f43b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:34.216629 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18039778dfde4bcaa1245544e3d3c1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:34.223634 4816 tablet_service.cc:1467] Processing CreateTablet for tablet f06f4eacdcec4b86a1bfd8e79be58bfe (DEFAULT_TABLE table=rebalance_test_table_4 [id=2f224bc51e0b4008a30114b6f180f43b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:34.224035 4816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f06f4eacdcec4b86a1bfd8e79be58bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:34.224134 4423 tablet_service.cc:1467] Processing CreateTablet for tablet f06f4eacdcec4b86a1bfd8e79be58bfe (DEFAULT_TABLE table=rebalance_test_table_4 [id=2f224bc51e0b4008a30114b6f180f43b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:34.224511 4423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f06f4eacdcec4b86a1bfd8e79be58bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:34.224758 5797 tablet_bootstrap.cc:492] T 18039778dfde4bcaa1245544e3d3c1a3 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:34.228487 5795 tablet_bootstrap.cc:492] T f06f4eacdcec4b86a1bfd8e79be58bfe P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:34.226414 5797 tablet_bootstrap.cc:654] T 18039778dfde4bcaa1245544e3d3c1a3 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:34.230300 5794 tablet_bootstrap.cc:492] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:34.230540 5795 tablet_bootstrap.cc:654] T f06f4eacdcec4b86a1bfd8e79be58bfe P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:34.232491 5794 tablet_bootstrap.cc:654] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:34.234411 5796 tablet_bootstrap.cc:492] T 18039778dfde4bcaa1245544e3d3c1a3 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:34.235324 5797 tablet_bootstrap.cc:492] T 18039778dfde4bcaa1245544e3d3c1a3 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:34.235507 5797 ts_tablet_manager.cc:1367] T 18039778dfde4bcaa1245544e3d3c1a3 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240502 21:26:34.236598 5796 tablet_bootstrap.cc:654] T 18039778dfde4bcaa1245544e3d3c1a3 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:34.237737 5797 raft_consensus.cc:348] T 18039778dfde4bcaa1245544e3d3c1a3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:34.238217 5797 raft_consensus.cc:374] T 18039778dfde4bcaa1245544e3d3c1a3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:34.238397 5797 raft_consensus.cc:729] T 18039778dfde4bcaa1245544e3d3c1a3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:34.238791 5797 consensus_queue.cc:260] T 18039778dfde4bcaa1245544e3d3c1a3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:34.241221 5795 tablet_bootstrap.cc:492] T f06f4eacdcec4b86a1bfd8e79be58bfe P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:34.241401 5795 ts_tablet_manager.cc:1367] T f06f4eacdcec4b86a1bfd8e79be58bfe P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240502 21:26:34.242429 5795 raft_consensus.cc:348] T f06f4eacdcec4b86a1bfd8e79be58bfe P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:34.242700 5795 raft_consensus.cc:374] T f06f4eacdcec4b86a1bfd8e79be58bfe P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:34.242811 5795 raft_consensus.cc:729] T f06f4eacdcec4b86a1bfd8e79be58bfe P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:34.243086 5795 consensus_queue.cc:260] T f06f4eacdcec4b86a1bfd8e79be58bfe P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:34.243760 5795 ts_tablet_manager.cc:1398] T f06f4eacdcec4b86a1bfd8e79be58bfe P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:26:34.244336 5797 ts_tablet_manager.cc:1398] T 18039778dfde4bcaa1245544e3d3c1a3 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:26:34.244625 5795 tablet_bootstrap.cc:492] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:34.246368 5795 tablet_bootstrap.cc:654] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:34.247366 5796 tablet_bootstrap.cc:492] T 18039778dfde4bcaa1245544e3d3c1a3 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:34.247509 5796 ts_tablet_manager.cc:1367] T 18039778dfde4bcaa1245544e3d3c1a3 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20240502 21:26:34.248221 5797 tablet_bootstrap.cc:492] T 20c21d242c8c4e00a45ae5de7388db3d P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:34.249006 5796 raft_consensus.cc:348] T 18039778dfde4bcaa1245544e3d3c1a3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:34.249207 5796 raft_consensus.cc:374] T 18039778dfde4bcaa1245544e3d3c1a3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:34.249295 5796 raft_consensus.cc:729] T 18039778dfde4bcaa1245544e3d3c1a3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:34.249497 5796 consensus_queue.cc:260] T 18039778dfde4bcaa1245544e3d3c1a3 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:34.250087 5796 ts_tablet_manager.cc:1398] T 18039778dfde4bcaa1245544e3d3c1a3 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:26:34.250347 5796 tablet_bootstrap.cc:492] T f06f4eacdcec4b86a1bfd8e79be58bfe P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:34.251829 5796 tablet_bootstrap.cc:654] T f06f4eacdcec4b86a1bfd8e79be58bfe P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:34.253388 5794 tablet_bootstrap.cc:492] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:34.253523 5794 ts_tablet_manager.cc:1367] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.009s
I20240502 21:26:34.254252 5794 raft_consensus.cc:348] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:34.254443 5794 raft_consensus.cc:374] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:34.254513 5794 raft_consensus.cc:729] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:34.254760 5794 consensus_queue.cc:260] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:34.255453 5794 ts_tablet_manager.cc:1398] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:34.255733 5794 tablet_bootstrap.cc:492] T 20c21d242c8c4e00a45ae5de7388db3d P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:34.257249 5794 tablet_bootstrap.cc:654] T 20c21d242c8c4e00a45ae5de7388db3d P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:34.260830 5795 tablet_bootstrap.cc:492] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:34.261021 5795 ts_tablet_manager.cc:1367] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.006s
I20240502 21:26:34.261919 5797 tablet_bootstrap.cc:654] T 20c21d242c8c4e00a45ae5de7388db3d P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:34.261940 5795 raft_consensus.cc:348] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:34.262219 5795 raft_consensus.cc:374] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:34.262310 5795 raft_consensus.cc:729] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:34.262605 5795 consensus_queue.cc:260] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:34.263247 5795 ts_tablet_manager.cc:1398] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 21:26:34.264055 5796 tablet_bootstrap.cc:492] T f06f4eacdcec4b86a1bfd8e79be58bfe P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:34.264219 5796 ts_tablet_manager.cc:1367] T f06f4eacdcec4b86a1bfd8e79be58bfe P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20240502 21:26:34.264796 5796 raft_consensus.cc:348] T f06f4eacdcec4b86a1bfd8e79be58bfe P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:34.264999 5796 raft_consensus.cc:374] T f06f4eacdcec4b86a1bfd8e79be58bfe P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:34.265084 5796 raft_consensus.cc:729] T f06f4eacdcec4b86a1bfd8e79be58bfe P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:34.265440 5797 tablet_bootstrap.cc:492] T 20c21d242c8c4e00a45ae5de7388db3d P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:34.265334 5796 consensus_queue.cc:260] T f06f4eacdcec4b86a1bfd8e79be58bfe P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:34.265908 5796 ts_tablet_manager.cc:1398] T f06f4eacdcec4b86a1bfd8e79be58bfe P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:26:34.266893 5794 tablet_bootstrap.cc:492] T 20c21d242c8c4e00a45ae5de7388db3d P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:34.267047 5794 ts_tablet_manager.cc:1367] T 20c21d242c8c4e00a45ae5de7388db3d P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.002s
I20240502 21:26:34.267385 5797 ts_tablet_manager.cc:1367] T 20c21d242c8c4e00a45ae5de7388db3d P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240502 21:26:34.267848 5794 raft_consensus.cc:348] T 20c21d242c8c4e00a45ae5de7388db3d P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:34.268102 5794 raft_consensus.cc:374] T 20c21d242c8c4e00a45ae5de7388db3d P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:34.268249 5794 raft_consensus.cc:729] T 20c21d242c8c4e00a45ae5de7388db3d P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:34.268581 5794 consensus_queue.cc:260] T 20c21d242c8c4e00a45ae5de7388db3d P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:34.268653 5797 raft_consensus.cc:348] T 20c21d242c8c4e00a45ae5de7388db3d P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:34.268955 5797 raft_consensus.cc:374] T 20c21d242c8c4e00a45ae5de7388db3d P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:34.269085 5797 raft_consensus.cc:729] T 20c21d242c8c4e00a45ae5de7388db3d P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:34.269153 5794 ts_tablet_manager.cc:1398] T 20c21d242c8c4e00a45ae5de7388db3d P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:34.269537 5794 tablet_bootstrap.cc:492] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:34.269618 5797 consensus_queue.cc:260] T 20c21d242c8c4e00a45ae5de7388db3d P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:34.271057 5794 tablet_bootstrap.cc:654] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:34.272382 5797 ts_tablet_manager.cc:1398] T 20c21d242c8c4e00a45ae5de7388db3d P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:26:34.273026 5794 tablet_bootstrap.cc:492] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:34.273165 5794 ts_tablet_manager.cc:1367] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 21:26:34.273887 5794 raft_consensus.cc:348] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:34.274040 5794 raft_consensus.cc:374] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:34.274137 5794 raft_consensus.cc:729] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:34.274360 5794 consensus_queue.cc:260] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:34.274832 5794 ts_tablet_manager.cc:1398] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:26:34.283785 5604 raft_consensus.cc:482] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:34.284006 5604 raft_consensus.cc:504] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:34.284667 5604 leader_election.cc:288] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:34.285254 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18039778dfde4bcaa1245544e3d3c1a3" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:26:34.285238 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18039778dfde4bcaa1245544e3d3c1a3" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:34.285523 4574 raft_consensus.cc:2454] T 18039778dfde4bcaa1245544e3d3c1a3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:26:34.285516 4838 raft_consensus.cc:2454] T 18039778dfde4bcaa1245544e3d3c1a3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:26:34.286046 4378 leader_election.cc:302] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:34.286386 5604 raft_consensus.cc:2781] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:34.286494 5604 raft_consensus.cc:482] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:34.286617 5604 raft_consensus.cc:3037] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:34.288909 5604 raft_consensus.cc:504] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:34.289572 5604 leader_election.cc:288] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:34.289979 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18039778dfde4bcaa1245544e3d3c1a3" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:26:34.290067 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18039778dfde4bcaa1245544e3d3c1a3" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:26:34.290230 4574 raft_consensus.cc:3037] T 18039778dfde4bcaa1245544e3d3c1a3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:34.290274 4838 raft_consensus.cc:3037] T 18039778dfde4bcaa1245544e3d3c1a3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:34.293128 4574 raft_consensus.cc:2454] T 18039778dfde4bcaa1245544e3d3c1a3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:26:34.293128 4838 raft_consensus.cc:2454] T 18039778dfde4bcaa1245544e3d3c1a3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:26:34.293612 4375 leader_election.cc:302] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:34.294142 5604 raft_consensus.cc:2781] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:34.294319 5604 raft_consensus.cc:686] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:26:34.294605 5604 consensus_queue.cc:237] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:34.297554 4293 catalog_manager.cc:5484] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:26:34.436105 5461 raft_consensus.cc:482] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:34.436328 5461 raft_consensus.cc:504] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:34.437050 5461 leader_election.cc:288] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:34.437597 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c21d242c8c4e00a45ae5de7388db3d" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:34.437654 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c21d242c8c4e00a45ae5de7388db3d" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:34.437856 4574 raft_consensus.cc:2454] T 20c21d242c8c4e00a45ae5de7388db3d P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:26:34.437917 4442 raft_consensus.cc:2454] T 20c21d242c8c4e00a45ae5de7388db3d P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:26:34.438350 4639 leader_election.cc:302] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:26:34.438637 5461 raft_consensus.cc:2781] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:34.438720 5461 raft_consensus.cc:482] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:34.438820 5461 raft_consensus.cc:3037] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:34.440812 5461 raft_consensus.cc:504] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:34.441341 5461 leader_election.cc:288] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Requested vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:34.441753 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c21d242c8c4e00a45ae5de7388db3d" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:26:34.441790 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c21d242c8c4e00a45ae5de7388db3d" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:34.441929 4574 raft_consensus.cc:3037] T 20c21d242c8c4e00a45ae5de7388db3d P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:34.441977 4442 raft_consensus.cc:3037] T 20c21d242c8c4e00a45ae5de7388db3d P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:34.443781 4574 raft_consensus.cc:2454] T 20c21d242c8c4e00a45ae5de7388db3d P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:26:34.443993 4442 raft_consensus.cc:2454] T 20c21d242c8c4e00a45ae5de7388db3d P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:26:34.444157 4642 leader_election.cc:302] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:34.444459 5461 raft_consensus.cc:2781] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:34.444854 5461 raft_consensus.cc:686] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Becoming Leader. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Running, Role: LEADER
I20240502 21:26:34.445106 5461 consensus_queue.cc:237] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:34.448084 4293 catalog_manager.cc:5484] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 398c3c8f3195492c900439ad5e1931bb (127.29.82.67). New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:26:34.527415 5604 raft_consensus.cc:482] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:34.527626 5604 raft_consensus.cc:504] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:34.528486 5604 leader_election.cc:288] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:34.528934 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f06f4eacdcec4b86a1bfd8e79be58bfe" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:26:34.528940 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f06f4eacdcec4b86a1bfd8e79be58bfe" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:34.529130 4838 raft_consensus.cc:2454] T f06f4eacdcec4b86a1bfd8e79be58bfe P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:26:34.529157 4705 raft_consensus.cc:2454] T f06f4eacdcec4b86a1bfd8e79be58bfe P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:26:34.529572 4375 leader_election.cc:302] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:34.529847 5604 raft_consensus.cc:2781] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:34.529945 5604 raft_consensus.cc:482] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:34.530011 5604 raft_consensus.cc:3037] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:34.531894 5604 raft_consensus.cc:504] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:34.532466 5604 leader_election.cc:288] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:34.532852 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f06f4eacdcec4b86a1bfd8e79be58bfe" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:34.532941 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f06f4eacdcec4b86a1bfd8e79be58bfe" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:26:34.533037 4705 raft_consensus.cc:3037] T f06f4eacdcec4b86a1bfd8e79be58bfe P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:34.533147 4838 raft_consensus.cc:3037] T f06f4eacdcec4b86a1bfd8e79be58bfe P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:34.535006 4838 raft_consensus.cc:2454] T f06f4eacdcec4b86a1bfd8e79be58bfe P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:26:34.535009 4705 raft_consensus.cc:2454] T f06f4eacdcec4b86a1bfd8e79be58bfe P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:26:34.535427 4375 leader_election.cc:302] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:34.535771 5604 raft_consensus.cc:2781] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:34.535915 5604 raft_consensus.cc:686] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:26:34.536180 5604 consensus_queue.cc:237] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:34.539048 4293 catalog_manager.cc:5484] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:26:34.746644 5604 consensus_queue.cc:1035] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:34.768198 5604 consensus_queue.cc:1035] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:34.944466 5811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:34.944801 5811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:34.952047 5811 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:26:34.957108 5461 consensus_queue.cc:1035] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:34.964151 5824 consensus_queue.cc:1035] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240502 21:26:35.029953 5604 consensus_queue.cc:1035] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:35.034847 5815 consensus_queue.cc:1035] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 230.531 ms
time per row: 0.115266 ms
I20240502 21:26:35.356901 4293 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:35.365617 4554 tablet_service.cc:1467] Processing CreateTablet for tablet e95d3785435e450688d87a0ce02d0a34 (DEFAULT_TABLE table=rebalance_test_table_5 [id=0b924d712d1446f6b103821acdc47c7b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:35.365957 4817 tablet_service.cc:1467] Processing CreateTablet for tablet e95d3785435e450688d87a0ce02d0a34 (DEFAULT_TABLE table=rebalance_test_table_5 [id=0b924d712d1446f6b103821acdc47c7b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:35.366153 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e95d3785435e450688d87a0ce02d0a34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:35.366155 4686 tablet_service.cc:1467] Processing CreateTablet for tablet e95d3785435e450688d87a0ce02d0a34 (DEFAULT_TABLE table=rebalance_test_table_5 [id=0b924d712d1446f6b103821acdc47c7b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:35.366436 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e95d3785435e450688d87a0ce02d0a34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:35.366603 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e95d3785435e450688d87a0ce02d0a34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:35.366568 4553 tablet_service.cc:1467] Processing CreateTablet for tablet dd1cf8e447e54d6e907ffa0e4681515f (DEFAULT_TABLE table=rebalance_test_table_5 [id=0b924d712d1446f6b103821acdc47c7b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:35.366922 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd1cf8e447e54d6e907ffa0e4681515f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:35.372284 4816 tablet_service.cc:1467] Processing CreateTablet for tablet dd1cf8e447e54d6e907ffa0e4681515f (DEFAULT_TABLE table=rebalance_test_table_5 [id=0b924d712d1446f6b103821acdc47c7b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:35.372639 4816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd1cf8e447e54d6e907ffa0e4681515f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:35.372557 4423 tablet_service.cc:1467] Processing CreateTablet for tablet 91044c2ac27d42f8a804bc5eadb13d8f (DEFAULT_TABLE table=rebalance_test_table_5 [id=0b924d712d1446f6b103821acdc47c7b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:35.372644 4555 tablet_service.cc:1467] Processing CreateTablet for tablet 91044c2ac27d42f8a804bc5eadb13d8f (DEFAULT_TABLE table=rebalance_test_table_5 [id=0b924d712d1446f6b103821acdc47c7b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:35.372987 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91044c2ac27d42f8a804bc5eadb13d8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:35.372983 4423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91044c2ac27d42f8a804bc5eadb13d8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:35.376185 4685 tablet_service.cc:1467] Processing CreateTablet for tablet dd1cf8e447e54d6e907ffa0e4681515f (DEFAULT_TABLE table=rebalance_test_table_5 [id=0b924d712d1446f6b103821acdc47c7b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:35.376595 4685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd1cf8e447e54d6e907ffa0e4681515f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:35.380597 4684 tablet_service.cc:1467] Processing CreateTablet for tablet 91044c2ac27d42f8a804bc5eadb13d8f (DEFAULT_TABLE table=rebalance_test_table_5 [id=0b924d712d1446f6b103821acdc47c7b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:35.381093 4684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91044c2ac27d42f8a804bc5eadb13d8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:35.390285 5877 tablet_bootstrap.cc:492] T e95d3785435e450688d87a0ce02d0a34 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:35.392524 5877 tablet_bootstrap.cc:654] T e95d3785435e450688d87a0ce02d0a34 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:35.394096 5874 tablet_bootstrap.cc:492] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:35.394156 5875 tablet_bootstrap.cc:492] T 91044c2ac27d42f8a804bc5eadb13d8f P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:35.394515 5876 tablet_bootstrap.cc:492] T e95d3785435e450688d87a0ce02d0a34 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:35.396190 5875 tablet_bootstrap.cc:654] T 91044c2ac27d42f8a804bc5eadb13d8f P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:35.396358 5876 tablet_bootstrap.cc:654] T e95d3785435e450688d87a0ce02d0a34 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:35.396379 5874 tablet_bootstrap.cc:654] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:35.398345 5877 tablet_bootstrap.cc:492] T e95d3785435e450688d87a0ce02d0a34 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:35.398494 5877 ts_tablet_manager.cc:1367] T e95d3785435e450688d87a0ce02d0a34 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240502 21:26:35.398666 5874 tablet_bootstrap.cc:492] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:35.398801 5874 ts_tablet_manager.cc:1367] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 21:26:35.399014 5876 tablet_bootstrap.cc:492] T e95d3785435e450688d87a0ce02d0a34 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:35.399158 5876 ts_tablet_manager.cc:1367] T e95d3785435e450688d87a0ce02d0a34 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:26:35.399341 5877 raft_consensus.cc:348] T e95d3785435e450688d87a0ce02d0a34 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:35.399552 5877 raft_consensus.cc:374] T e95d3785435e450688d87a0ce02d0a34 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:35.399492 5874 raft_consensus.cc:348] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:35.399693 5877 raft_consensus.cc:729] T e95d3785435e450688d87a0ce02d0a34 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:35.399760 5874 raft_consensus.cc:374] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:35.399909 5874 raft_consensus.cc:729] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:35.399940 5877 consensus_queue.cc:260] T e95d3785435e450688d87a0ce02d0a34 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:35.399974 5876 raft_consensus.cc:348] T e95d3785435e450688d87a0ce02d0a34 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:35.400189 5876 raft_consensus.cc:374] T e95d3785435e450688d87a0ce02d0a34 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:35.400161 5874 consensus_queue.cc:260] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:35.400315 5876 raft_consensus.cc:729] T e95d3785435e450688d87a0ce02d0a34 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:35.400615 5877 ts_tablet_manager.cc:1398] T e95d3785435e450688d87a0ce02d0a34 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 21:26:35.400902 5874 ts_tablet_manager.cc:1398] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:35.400969 5877 tablet_bootstrap.cc:492] T dd1cf8e447e54d6e907ffa0e4681515f P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:35.401167 5874 tablet_bootstrap.cc:492] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:35.402436 5877 tablet_bootstrap.cc:654] T dd1cf8e447e54d6e907ffa0e4681515f P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:35.402824 5874 tablet_bootstrap.cc:654] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:35.406558 5876 consensus_queue.cc:260] T e95d3785435e450688d87a0ce02d0a34 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:35.407912 5876 ts_tablet_manager.cc:1398] T e95d3785435e450688d87a0ce02d0a34 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240502 21:26:35.408272 5876 tablet_bootstrap.cc:492] T dd1cf8e447e54d6e907ffa0e4681515f P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:35.408821 5875 tablet_bootstrap.cc:492] T 91044c2ac27d42f8a804bc5eadb13d8f P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:35.408984 5875 ts_tablet_manager.cc:1367] T 91044c2ac27d42f8a804bc5eadb13d8f P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240502 21:26:35.409849 5875 raft_consensus.cc:348] T 91044c2ac27d42f8a804bc5eadb13d8f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:35.410077 5875 raft_consensus.cc:374] T 91044c2ac27d42f8a804bc5eadb13d8f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:35.410107 5876 tablet_bootstrap.cc:654] T dd1cf8e447e54d6e907ffa0e4681515f P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:35.410176 5875 raft_consensus.cc:729] T 91044c2ac27d42f8a804bc5eadb13d8f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:35.410419 5877 tablet_bootstrap.cc:492] T dd1cf8e447e54d6e907ffa0e4681515f P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:35.410423 5875 consensus_queue.cc:260] T 91044c2ac27d42f8a804bc5eadb13d8f P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:35.410588 5877 ts_tablet_manager.cc:1367] T dd1cf8e447e54d6e907ffa0e4681515f P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240502 21:26:35.411492 5877 raft_consensus.cc:348] T dd1cf8e447e54d6e907ffa0e4681515f P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:35.411835 5877 raft_consensus.cc:374] T dd1cf8e447e54d6e907ffa0e4681515f P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:35.411931 5877 raft_consensus.cc:729] T dd1cf8e447e54d6e907ffa0e4681515f P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:35.412283 5877 consensus_queue.cc:260] T dd1cf8e447e54d6e907ffa0e4681515f P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:35.412547 5875 ts_tablet_manager.cc:1398] T 91044c2ac27d42f8a804bc5eadb13d8f P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:35.413841 5877 ts_tablet_manager.cc:1398] T dd1cf8e447e54d6e907ffa0e4681515f P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:35.414211 5877 tablet_bootstrap.cc:492] T 91044c2ac27d42f8a804bc5eadb13d8f P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:35.414680 5874 tablet_bootstrap.cc:492] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:35.414887 5874 ts_tablet_manager.cc:1367] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.001s
I20240502 21:26:35.415755 5874 raft_consensus.cc:348] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:35.416018 5874 raft_consensus.cc:374] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:35.416127 5874 raft_consensus.cc:729] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:35.416450 5874 consensus_queue.cc:260] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:35.417330 5877 tablet_bootstrap.cc:654] T 91044c2ac27d42f8a804bc5eadb13d8f P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:35.418007 5874 ts_tablet_manager.cc:1398] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:35.422369 5877 tablet_bootstrap.cc:492] T 91044c2ac27d42f8a804bc5eadb13d8f P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:35.422521 5877 ts_tablet_manager.cc:1367] T 91044c2ac27d42f8a804bc5eadb13d8f P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 21:26:35.422756 5876 tablet_bootstrap.cc:492] T dd1cf8e447e54d6e907ffa0e4681515f P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:35.422946 5876 ts_tablet_manager.cc:1367] T dd1cf8e447e54d6e907ffa0e4681515f P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20240502 21:26:35.423384 5877 raft_consensus.cc:348] T 91044c2ac27d42f8a804bc5eadb13d8f P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:35.423612 5877 raft_consensus.cc:374] T 91044c2ac27d42f8a804bc5eadb13d8f P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:35.423717 5877 raft_consensus.cc:729] T 91044c2ac27d42f8a804bc5eadb13d8f P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:35.423828 5876 raft_consensus.cc:348] T dd1cf8e447e54d6e907ffa0e4681515f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:35.423967 5877 consensus_queue.cc:260] T 91044c2ac27d42f8a804bc5eadb13d8f P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:35.424091 5876 raft_consensus.cc:374] T dd1cf8e447e54d6e907ffa0e4681515f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:35.424163 5876 raft_consensus.cc:729] T dd1cf8e447e54d6e907ffa0e4681515f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:35.424536 5877 ts_tablet_manager.cc:1398] T 91044c2ac27d42f8a804bc5eadb13d8f P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:35.424432 5876 consensus_queue.cc:260] T dd1cf8e447e54d6e907ffa0e4681515f P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:35.425148 5876 ts_tablet_manager.cc:1398] T dd1cf8e447e54d6e907ffa0e4681515f P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:26:35.425433 5876 tablet_bootstrap.cc:492] T 91044c2ac27d42f8a804bc5eadb13d8f P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:35.426618 5876 tablet_bootstrap.cc:654] T 91044c2ac27d42f8a804bc5eadb13d8f P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:35.428467 5876 tablet_bootstrap.cc:492] T 91044c2ac27d42f8a804bc5eadb13d8f P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:35.428622 5876 ts_tablet_manager.cc:1367] T 91044c2ac27d42f8a804bc5eadb13d8f P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:35.429404 5876 raft_consensus.cc:348] T 91044c2ac27d42f8a804bc5eadb13d8f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:35.429586 5876 raft_consensus.cc:374] T 91044c2ac27d42f8a804bc5eadb13d8f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:35.429674 5876 raft_consensus.cc:729] T 91044c2ac27d42f8a804bc5eadb13d8f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:35.429875 5876 consensus_queue.cc:260] T 91044c2ac27d42f8a804bc5eadb13d8f P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:35.430397 5876 ts_tablet_manager.cc:1398] T 91044c2ac27d42f8a804bc5eadb13d8f P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:35.510236 5783 raft_consensus.cc:482] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:35.510416 5783 raft_consensus.cc:504] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:35.511018 5783 leader_election.cc:288] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:35.511374 5783 raft_consensus.cc:482] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:35.511548 5783 raft_consensus.cc:504] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:35.511628 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e95d3785435e450688d87a0ce02d0a34" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:35.511663 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e95d3785435e450688d87a0ce02d0a34" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:35.511894 4705 raft_consensus.cc:2454] T e95d3785435e450688d87a0ce02d0a34 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:35.511912 4574 raft_consensus.cc:2454] T e95d3785435e450688d87a0ce02d0a34 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:35.512207 5783 leader_election.cc:288] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:35.512468 4773 leader_election.cc:302] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:35.512593 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd1cf8e447e54d6e907ffa0e4681515f" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:35.512760 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd1cf8e447e54d6e907ffa0e4681515f" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:35.512825 4705 raft_consensus.cc:2454] T dd1cf8e447e54d6e907ffa0e4681515f P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:35.512975 4574 raft_consensus.cc:2454] T dd1cf8e447e54d6e907ffa0e4681515f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:35.513006 5783 raft_consensus.cc:2781] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:35.513103 5783 raft_consensus.cc:482] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:35.513235 5783 raft_consensus.cc:3037] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:35.513279 4773 leader_election.cc:302] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:35.513578 5732 raft_consensus.cc:2781] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:35.513718 5732 raft_consensus.cc:482] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:35.513846 5732 raft_consensus.cc:3037] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:35.515689 5783 raft_consensus.cc:504] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:35.515915 5732 raft_consensus.cc:504] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:35.516238 5783 leader_election.cc:288] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:35.516563 5732 leader_election.cc:288] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:35.516764 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd1cf8e447e54d6e907ffa0e4681515f" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:35.516837 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e95d3785435e450688d87a0ce02d0a34" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:26:35.516947 4706 raft_consensus.cc:3037] T dd1cf8e447e54d6e907ffa0e4681515f P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:35.516923 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd1cf8e447e54d6e907ffa0e4681515f" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:26:35.517055 4574 raft_consensus.cc:3037] T e95d3785435e450688d87a0ce02d0a34 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:35.517124 4575 raft_consensus.cc:3037] T dd1cf8e447e54d6e907ffa0e4681515f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:35.517524 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e95d3785435e450688d87a0ce02d0a34" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:35.517758 4705 raft_consensus.cc:3037] T e95d3785435e450688d87a0ce02d0a34 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:35.519364 4706 raft_consensus.cc:2454] T dd1cf8e447e54d6e907ffa0e4681515f P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:35.519594 4575 raft_consensus.cc:2454] T dd1cf8e447e54d6e907ffa0e4681515f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:35.519732 4773 leader_election.cc:302] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:35.520011 5732 raft_consensus.cc:2781] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:35.520179 5732 raft_consensus.cc:686] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:26:35.520457 5732 consensus_queue.cc:237] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:35.523312 4293 catalog_manager.cc:5484] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:26:35.549700 4705 raft_consensus.cc:2454] T e95d3785435e450688d87a0ce02d0a34 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:35.549973 4574 raft_consensus.cc:2454] T e95d3785435e450688d87a0ce02d0a34 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:35.550207 4773 leader_election.cc:302] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:35.550544 5732 raft_consensus.cc:2781] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:35.550685 5732 raft_consensus.cc:686] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:26:35.550938 5732 consensus_queue.cc:237] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:35.553915 4293 catalog_manager.cc:5484] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:26:35.674983 5602 raft_consensus.cc:482] T 91044c2ac27d42f8a804bc5eadb13d8f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:35.675174 5602 raft_consensus.cc:504] T 91044c2ac27d42f8a804bc5eadb13d8f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:35.675848 5602 leader_election.cc:288] T 91044c2ac27d42f8a804bc5eadb13d8f P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:35.676239 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91044c2ac27d42f8a804bc5eadb13d8f" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:35.676453 4442 raft_consensus.cc:2454] T 91044c2ac27d42f8a804bc5eadb13d8f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:26:35.676452 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91044c2ac27d42f8a804bc5eadb13d8f" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:35.676723 4706 raft_consensus.cc:2454] T 91044c2ac27d42f8a804bc5eadb13d8f P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:26:35.676882 4508 leader_election.cc:302] T 91044c2ac27d42f8a804bc5eadb13d8f P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:35.677209 5602 raft_consensus.cc:2781] T 91044c2ac27d42f8a804bc5eadb13d8f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:35.677302 5602 raft_consensus.cc:482] T 91044c2ac27d42f8a804bc5eadb13d8f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:35.677405 5602 raft_consensus.cc:3037] T 91044c2ac27d42f8a804bc5eadb13d8f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:35.679344 5602 raft_consensus.cc:504] T 91044c2ac27d42f8a804bc5eadb13d8f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:35.679934 5602 leader_election.cc:288] T 91044c2ac27d42f8a804bc5eadb13d8f P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:35.680372 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91044c2ac27d42f8a804bc5eadb13d8f" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:35.680459 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91044c2ac27d42f8a804bc5eadb13d8f" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:35.680575 4442 raft_consensus.cc:3037] T 91044c2ac27d42f8a804bc5eadb13d8f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:35.680680 4706 raft_consensus.cc:3037] T 91044c2ac27d42f8a804bc5eadb13d8f P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:35.682461 4442 raft_consensus.cc:2454] T 91044c2ac27d42f8a804bc5eadb13d8f P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:26:35.682667 4706 raft_consensus.cc:2454] T 91044c2ac27d42f8a804bc5eadb13d8f P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:26:35.682876 4508 leader_election.cc:302] T 91044c2ac27d42f8a804bc5eadb13d8f P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:35.683198 5602 raft_consensus.cc:2781] T 91044c2ac27d42f8a804bc5eadb13d8f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:35.683727 5602 raft_consensus.cc:686] T 91044c2ac27d42f8a804bc5eadb13d8f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:26:35.684367 5602 consensus_queue.cc:237] T 91044c2ac27d42f8a804bc5eadb13d8f P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:35.687405 4293 catalog_manager.cc:5484] T 91044c2ac27d42f8a804bc5eadb13d8f P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:26:35.929890 5783 consensus_queue.cc:1035] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:35.936705 5783 consensus_queue.cc:1035] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:36.041729 5783 consensus_queue.cc:1035] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:36.047490 5732 consensus_queue.cc:1035] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:36.085944 5892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:36.086354 5892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:36.094748 5892 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 21:26:36.243247 4443 raft_consensus.cc:1261] T 91044c2ac27d42f8a804bc5eadb13d8f P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:36.244264 5890 consensus_queue.cc:1035] T 91044c2ac27d42f8a804bc5eadb13d8f P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:36.264261 4705 raft_consensus.cc:1261] T 91044c2ac27d42f8a804bc5eadb13d8f P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:36.300772 5890 consensus_queue.cc:1035] T 91044c2ac27d42f8a804bc5eadb13d8f P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 286.476 ms
time per row: 0.143238 ms
I20240502 21:26:36.545104 4295 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:36.553761 4817 tablet_service.cc:1467] Processing CreateTablet for tablet c856714eb4ea454d818d467d84757eb1 (DEFAULT_TABLE table=rebalance_test_table_6 [id=4cf0231801934a10bad256a243b59d8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:36.554127 4423 tablet_service.cc:1467] Processing CreateTablet for tablet c856714eb4ea454d818d467d84757eb1 (DEFAULT_TABLE table=rebalance_test_table_6 [id=4cf0231801934a10bad256a243b59d8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:36.554286 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c856714eb4ea454d818d467d84757eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:36.554651 4423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c856714eb4ea454d818d467d84757eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:36.555133 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 06fa2015dc7b4d97bf7e00833e5fa4b6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=4cf0231801934a10bad256a243b59d8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:36.555605 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06fa2015dc7b4d97bf7e00833e5fa4b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:36.556406 4555 tablet_service.cc:1467] Processing CreateTablet for tablet ccedd81bdfda43fbac7df75535af5cb6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=4cf0231801934a10bad256a243b59d8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:36.556828 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccedd81bdfda43fbac7df75535af5cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:36.558415 4816 tablet_service.cc:1467] Processing CreateTablet for tablet 06fa2015dc7b4d97bf7e00833e5fa4b6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=4cf0231801934a10bad256a243b59d8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:36.558761 4816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06fa2015dc7b4d97bf7e00833e5fa4b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:36.560297 4554 tablet_service.cc:1467] Processing CreateTablet for tablet c856714eb4ea454d818d467d84757eb1 (DEFAULT_TABLE table=rebalance_test_table_6 [id=4cf0231801934a10bad256a243b59d8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:36.560654 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c856714eb4ea454d818d467d84757eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:36.562573 4818 tablet_service.cc:1467] Processing CreateTablet for tablet ccedd81bdfda43fbac7df75535af5cb6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=4cf0231801934a10bad256a243b59d8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:36.562985 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccedd81bdfda43fbac7df75535af5cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:36.570086 4421 tablet_service.cc:1467] Processing CreateTablet for tablet 06fa2015dc7b4d97bf7e00833e5fa4b6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=4cf0231801934a10bad256a243b59d8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:36.571004 4422 tablet_service.cc:1467] Processing CreateTablet for tablet ccedd81bdfda43fbac7df75535af5cb6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=4cf0231801934a10bad256a243b59d8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:36.572419 5959 tablet_bootstrap.cc:492] T c856714eb4ea454d818d467d84757eb1 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:36.574311 5959 tablet_bootstrap.cc:654] T c856714eb4ea454d818d467d84757eb1 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:36.577591 5960 tablet_bootstrap.cc:492] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:36.579656 5958 tablet_bootstrap.cc:492] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:36.581557 5958 tablet_bootstrap.cc:654] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:36.581908 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06fa2015dc7b4d97bf7e00833e5fa4b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:36.582250 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccedd81bdfda43fbac7df75535af5cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:36.583329 5959 tablet_bootstrap.cc:492] T c856714eb4ea454d818d467d84757eb1 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:36.583932 5960 tablet_bootstrap.cc:654] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:36.584196 5959 ts_tablet_manager.cc:1367] T c856714eb4ea454d818d467d84757eb1 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20240502 21:26:36.585131 5959 raft_consensus.cc:348] T c856714eb4ea454d818d467d84757eb1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:36.585479 5959 raft_consensus.cc:374] T c856714eb4ea454d818d467d84757eb1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:36.585613 5959 raft_consensus.cc:729] T c856714eb4ea454d818d467d84757eb1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:36.585971 5959 consensus_queue.cc:260] T c856714eb4ea454d818d467d84757eb1 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:36.586704 5959 ts_tablet_manager.cc:1398] T c856714eb4ea454d818d467d84757eb1 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:36.587126 5959 tablet_bootstrap.cc:492] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:36.589700 5959 tablet_bootstrap.cc:654] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:36.590831 5962 tablet_bootstrap.cc:492] T c856714eb4ea454d818d467d84757eb1 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:36.592831 5962 tablet_bootstrap.cc:654] T c856714eb4ea454d818d467d84757eb1 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:36.594228 5959 tablet_bootstrap.cc:492] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:36.594249 5960 tablet_bootstrap.cc:492] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:36.594388 5960 ts_tablet_manager.cc:1367] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20240502 21:26:36.594408 5959 ts_tablet_manager.cc:1367] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 21:26:36.595266 5959 raft_consensus.cc:348] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:36.595314 5960 raft_consensus.cc:348] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:36.595501 5960 raft_consensus.cc:374] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:36.595604 5959 raft_consensus.cc:374] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:36.595667 5958 tablet_bootstrap.cc:492] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:36.595695 5959 raft_consensus.cc:729] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:36.595729 5960 raft_consensus.cc:729] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:36.595927 5958 ts_tablet_manager.cc:1367] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.002s
I20240502 21:26:36.595985 5959 consensus_queue.cc:260] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:36.596139 5960 consensus_queue.cc:260] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:36.596930 5960 ts_tablet_manager.cc:1398] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:36.597218 5960 tablet_bootstrap.cc:492] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:36.600519 5960 tablet_bootstrap.cc:654] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:36.601091 5958 raft_consensus.cc:348] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:36.601449 5958 raft_consensus.cc:374] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:36.601609 5958 raft_consensus.cc:729] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:36.601963 5958 consensus_queue.cc:260] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:36.602743 5958 ts_tablet_manager.cc:1398] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 21:26:36.603278 5959 ts_tablet_manager.cc:1398] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240502 21:26:36.607085 5959 tablet_bootstrap.cc:492] T ccedd81bdfda43fbac7df75535af5cb6 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:36.609457 5959 tablet_bootstrap.cc:654] T ccedd81bdfda43fbac7df75535af5cb6 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:36.610134 5960 tablet_bootstrap.cc:492] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:36.610309 5960 ts_tablet_manager.cc:1367] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.006s
I20240502 21:26:36.611050 5960 raft_consensus.cc:348] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:36.611228 5960 raft_consensus.cc:374] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:36.611312 5960 raft_consensus.cc:729] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:36.611652 5960 consensus_queue.cc:260] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:36.611799 5962 tablet_bootstrap.cc:492] T c856714eb4ea454d818d467d84757eb1 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:36.611982 5962 ts_tablet_manager.cc:1367] T c856714eb4ea454d818d467d84757eb1 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.015s
I20240502 21:26:36.612262 5960 ts_tablet_manager.cc:1398] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:36.612285 5959 tablet_bootstrap.cc:492] T ccedd81bdfda43fbac7df75535af5cb6 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:36.612421 5959 ts_tablet_manager.cc:1367] T ccedd81bdfda43fbac7df75535af5cb6 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240502 21:26:36.612962 5962 raft_consensus.cc:348] T c856714eb4ea454d818d467d84757eb1 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:36.613186 5962 raft_consensus.cc:374] T c856714eb4ea454d818d467d84757eb1 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:36.613348 5962 raft_consensus.cc:729] T c856714eb4ea454d818d467d84757eb1 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:36.613291 5959 raft_consensus.cc:348] T ccedd81bdfda43fbac7df75535af5cb6 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:36.613502 5959 raft_consensus.cc:374] T ccedd81bdfda43fbac7df75535af5cb6 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:36.613677 5959 raft_consensus.cc:729] T ccedd81bdfda43fbac7df75535af5cb6 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:36.613588 5962 consensus_queue.cc:260] T c856714eb4ea454d818d467d84757eb1 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:36.614171 5962 ts_tablet_manager.cc:1398] T c856714eb4ea454d818d467d84757eb1 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:36.614464 5962 tablet_bootstrap.cc:492] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:36.613946 5959 consensus_queue.cc:260] T ccedd81bdfda43fbac7df75535af5cb6 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:36.616171 5962 tablet_bootstrap.cc:654] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:36.617645 5959 ts_tablet_manager.cc:1398] T ccedd81bdfda43fbac7df75535af5cb6 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:26:36.618391 5962 tablet_bootstrap.cc:492] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:36.619258 5962 ts_tablet_manager.cc:1367] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 21:26:36.620283 5962 raft_consensus.cc:348] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:36.620515 5962 raft_consensus.cc:374] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:36.620605 5962 raft_consensus.cc:729] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:36.620858 5962 consensus_queue.cc:260] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:36.621377 5962 ts_tablet_manager.cc:1398] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:26:36.621685 5962 tablet_bootstrap.cc:492] T ccedd81bdfda43fbac7df75535af5cb6 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:36.622895 5962 tablet_bootstrap.cc:654] T ccedd81bdfda43fbac7df75535af5cb6 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:36.624747 5962 tablet_bootstrap.cc:492] T ccedd81bdfda43fbac7df75535af5cb6 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:36.624895 5962 ts_tablet_manager.cc:1367] T ccedd81bdfda43fbac7df75535af5cb6 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20240502 21:26:36.625478 5962 raft_consensus.cc:348] T ccedd81bdfda43fbac7df75535af5cb6 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:36.625628 5962 raft_consensus.cc:374] T ccedd81bdfda43fbac7df75535af5cb6 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:36.625726 5962 raft_consensus.cc:729] T ccedd81bdfda43fbac7df75535af5cb6 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:36.625931 5962 consensus_queue.cc:260] T ccedd81bdfda43fbac7df75535af5cb6 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:36.626343 5962 ts_tablet_manager.cc:1398] T ccedd81bdfda43fbac7df75535af5cb6 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:26:36.638201 5890 raft_consensus.cc:482] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:36.638358 5890 raft_consensus.cc:504] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:36.638998 5890 leader_election.cc:288] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:36.639467 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccedd81bdfda43fbac7df75535af5cb6" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:36.639431 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccedd81bdfda43fbac7df75535af5cb6" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:26:36.639693 4443 raft_consensus.cc:2454] T ccedd81bdfda43fbac7df75535af5cb6 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:26:36.639707 4839 raft_consensus.cc:2454] T ccedd81bdfda43fbac7df75535af5cb6 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:26:36.640105 4508 leader_election.cc:302] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:36.640362 5890 raft_consensus.cc:2781] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:36.640480 5890 raft_consensus.cc:482] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:36.640571 5890 raft_consensus.cc:3037] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:36.642392 5890 raft_consensus.cc:504] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:36.642858 5890 leader_election.cc:288] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Requested vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:36.643311 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccedd81bdfda43fbac7df75535af5cb6" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:36.643307 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccedd81bdfda43fbac7df75535af5cb6" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:26:36.643478 4443 raft_consensus.cc:3037] T ccedd81bdfda43fbac7df75535af5cb6 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:36.643481 4839 raft_consensus.cc:3037] T ccedd81bdfda43fbac7df75535af5cb6 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:36.645277 4443 raft_consensus.cc:2454] T ccedd81bdfda43fbac7df75535af5cb6 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:26:36.645382 4839 raft_consensus.cc:2454] T ccedd81bdfda43fbac7df75535af5cb6 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:26:36.645713 4508 leader_election.cc:302] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:36.646008 5890 raft_consensus.cc:2781] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:36.646162 5890 raft_consensus.cc:686] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:26:36.646446 5890 consensus_queue.cc:237] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:36.649235 4293 catalog_manager.cc:5484] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:26:36.669770 5890 raft_consensus.cc:482] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:36.669984 5890 raft_consensus.cc:504] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:36.670596 5890 leader_election.cc:288] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:36.671070 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c856714eb4ea454d818d467d84757eb1" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:26:36.671129 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c856714eb4ea454d818d467d84757eb1" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:36.671272 4839 raft_consensus.cc:2454] T c856714eb4ea454d818d467d84757eb1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:26:36.671489 4443 raft_consensus.cc:2454] T c856714eb4ea454d818d467d84757eb1 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:26:36.671677 4508 leader_election.cc:302] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:36.671940 5890 raft_consensus.cc:2781] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:36.672045 5890 raft_consensus.cc:482] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:36.672153 5890 raft_consensus.cc:3037] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:36.674006 5890 raft_consensus.cc:504] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:36.674513 5890 leader_election.cc:288] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Requested vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:36.674939 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c856714eb4ea454d818d467d84757eb1" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:26:36.674983 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c856714eb4ea454d818d467d84757eb1" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:36.675120 4839 raft_consensus.cc:3037] T c856714eb4ea454d818d467d84757eb1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:36.675165 4443 raft_consensus.cc:3037] T c856714eb4ea454d818d467d84757eb1 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:36.677070 4443 raft_consensus.cc:2454] T c856714eb4ea454d818d467d84757eb1 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:26:36.677145 4839 raft_consensus.cc:2454] T c856714eb4ea454d818d467d84757eb1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:26:36.677515 4508 leader_election.cc:302] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:36.677775 5890 raft_consensus.cc:2781] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:36.677932 5890 raft_consensus.cc:686] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:26:36.678211 5890 consensus_queue.cc:237] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:36.681255 4293 catalog_manager.cc:5484] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:26:36.932240 5723 raft_consensus.cc:482] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:36.932471 5723 raft_consensus.cc:504] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:36.933125 5723 leader_election.cc:288] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:36.933554 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06fa2015dc7b4d97bf7e00833e5fa4b6" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:26:36.933591 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06fa2015dc7b4d97bf7e00833e5fa4b6" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:36.933790 4839 raft_consensus.cc:2454] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:26:36.933830 4706 raft_consensus.cc:2454] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:26:36.934238 4378 leader_election.cc:302] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:36.934557 5723 raft_consensus.cc:2781] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:36.934648 5723 raft_consensus.cc:482] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:36.934752 5723 raft_consensus.cc:3037] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:36.936777 5723 raft_consensus.cc:504] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:36.937274 5723 leader_election.cc:288] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:36.937682 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06fa2015dc7b4d97bf7e00833e5fa4b6" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:26:36.937737 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06fa2015dc7b4d97bf7e00833e5fa4b6" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:36.937918 4839 raft_consensus.cc:3037] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:36.937930 4706 raft_consensus.cc:3037] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:36.940692 4839 raft_consensus.cc:2454] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:26:36.940696 4706 raft_consensus.cc:2454] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:26:36.941059 4378 leader_election.cc:302] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:26:36.941300 5723 raft_consensus.cc:2781] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:36.941617 5723 raft_consensus.cc:686] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:26:36.941890 5723 consensus_queue.cc:237] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:36.944434 4293 catalog_manager.cc:5484] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:26:37.064944 5890 consensus_queue.cc:1035] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:37.073315 5890 consensus_queue.cc:1035] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:37.145239 5890 consensus_queue.cc:1035] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:37.154261 5936 consensus_queue.cc:1035] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:37.382366 5975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:37.382737 5975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:37.390915 5975 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240502 21:26:37.481317 5723 consensus_queue.cc:1035] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:37.530012 6007 consensus_queue.cc:1035] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:37.576859 4375 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.68:42449, user_credentials={real_user=slave}} blocked reactor thread for 67486.4us
INSERT report
rows total: 2000
time total: 212.892 ms
time per row: 0.106446 ms
I20240502 21:26:37.763814 4295 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:37.773586 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 2285212d78b0460ca329949eb3140e69 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ccfdd338c3a14e4eaab745d78ee6de47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:37.773979 4555 tablet_service.cc:1467] Processing CreateTablet for tablet 2285212d78b0460ca329949eb3140e69 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ccfdd338c3a14e4eaab745d78ee6de47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:37.774116 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2285212d78b0460ca329949eb3140e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:37.774402 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2285212d78b0460ca329949eb3140e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:37.774343 4422 tablet_service.cc:1467] Processing CreateTablet for tablet 2285212d78b0460ca329949eb3140e69 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ccfdd338c3a14e4eaab745d78ee6de47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:37.774784 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2285212d78b0460ca329949eb3140e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:37.776377 4818 tablet_service.cc:1467] Processing CreateTablet for tablet 9a3c115189e8442ba7c3fa7f65089d11 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ccfdd338c3a14e4eaab745d78ee6de47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:37.776645 4685 tablet_service.cc:1467] Processing CreateTablet for tablet a33fd7d5a8db41a39ba780e16a84f9b1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ccfdd338c3a14e4eaab745d78ee6de47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:37.776808 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a3c115189e8442ba7c3fa7f65089d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:37.777014 4685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a33fd7d5a8db41a39ba780e16a84f9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:37.777315 4684 tablet_service.cc:1467] Processing CreateTablet for tablet 9a3c115189e8442ba7c3fa7f65089d11 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ccfdd338c3a14e4eaab745d78ee6de47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:37.777668 4684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a3c115189e8442ba7c3fa7f65089d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:37.780556 4817 tablet_service.cc:1467] Processing CreateTablet for tablet a33fd7d5a8db41a39ba780e16a84f9b1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ccfdd338c3a14e4eaab745d78ee6de47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:37.780700 4423 tablet_service.cc:1467] Processing CreateTablet for tablet 9a3c115189e8442ba7c3fa7f65089d11 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ccfdd338c3a14e4eaab745d78ee6de47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:37.780931 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a33fd7d5a8db41a39ba780e16a84f9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:37.781090 4423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a3c115189e8442ba7c3fa7f65089d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:37.785861 4554 tablet_service.cc:1467] Processing CreateTablet for tablet a33fd7d5a8db41a39ba780e16a84f9b1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ccfdd338c3a14e4eaab745d78ee6de47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:37.786404 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a33fd7d5a8db41a39ba780e16a84f9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:37.804318 6047 tablet_bootstrap.cc:492] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:37.806634 6047 tablet_bootstrap.cc:654] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:37.808422 6045 tablet_bootstrap.cc:492] T 2285212d78b0460ca329949eb3140e69 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:37.812569 6048 tablet_bootstrap.cc:492] T 2285212d78b0460ca329949eb3140e69 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:37.812600 6045 tablet_bootstrap.cc:654] T 2285212d78b0460ca329949eb3140e69 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:37.812587 6046 tablet_bootstrap.cc:492] T 9a3c115189e8442ba7c3fa7f65089d11 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:37.814461 6047 tablet_bootstrap.cc:492] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:37.814528 6046 tablet_bootstrap.cc:654] T 9a3c115189e8442ba7c3fa7f65089d11 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:37.814519 6048 tablet_bootstrap.cc:654] T 2285212d78b0460ca329949eb3140e69 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:37.814658 6045 tablet_bootstrap.cc:492] T 2285212d78b0460ca329949eb3140e69 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:37.814661 6047 ts_tablet_manager.cc:1367] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.005s
I20240502 21:26:37.814875 6045 ts_tablet_manager.cc:1367] T 2285212d78b0460ca329949eb3140e69 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 21:26:37.815490 6047 raft_consensus.cc:348] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:37.815776 6047 raft_consensus.cc:374] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:37.815907 6047 raft_consensus.cc:729] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:37.815708 6045 raft_consensus.cc:348] T 2285212d78b0460ca329949eb3140e69 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:37.816063 6045 raft_consensus.cc:374] T 2285212d78b0460ca329949eb3140e69 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:37.816211 6045 raft_consensus.cc:729] T 2285212d78b0460ca329949eb3140e69 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:37.816223 6047 consensus_queue.cc:260] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:37.816608 6045 consensus_queue.cc:260] T 2285212d78b0460ca329949eb3140e69 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:37.816849 6047 ts_tablet_manager.cc:1398] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:37.817373 6045 ts_tablet_manager.cc:1398] T 2285212d78b0460ca329949eb3140e69 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:37.817690 6045 tablet_bootstrap.cc:492] T 9a3c115189e8442ba7c3fa7f65089d11 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:37.818199 6047 tablet_bootstrap.cc:492] T a33fd7d5a8db41a39ba780e16a84f9b1 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:37.819989 6047 tablet_bootstrap.cc:654] T a33fd7d5a8db41a39ba780e16a84f9b1 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:37.824461 6045 tablet_bootstrap.cc:654] T 9a3c115189e8442ba7c3fa7f65089d11 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:37.830085 6048 tablet_bootstrap.cc:492] T 2285212d78b0460ca329949eb3140e69 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:37.830302 6048 ts_tablet_manager.cc:1367] T 2285212d78b0460ca329949eb3140e69 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.004s
I20240502 21:26:37.830636 6046 tablet_bootstrap.cc:492] T 9a3c115189e8442ba7c3fa7f65089d11 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:37.831226 6046 ts_tablet_manager.cc:1367] T 9a3c115189e8442ba7c3fa7f65089d11 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.011s
I20240502 21:26:37.831261 6048 raft_consensus.cc:348] T 2285212d78b0460ca329949eb3140e69 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:37.831620 6048 raft_consensus.cc:374] T 2285212d78b0460ca329949eb3140e69 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:37.831784 6045 tablet_bootstrap.cc:492] T 9a3c115189e8442ba7c3fa7f65089d11 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:37.831784 6048 raft_consensus.cc:729] T 2285212d78b0460ca329949eb3140e69 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:37.831981 6047 tablet_bootstrap.cc:492] T a33fd7d5a8db41a39ba780e16a84f9b1 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:37.832046 6045 ts_tablet_manager.cc:1367] T 9a3c115189e8442ba7c3fa7f65089d11 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.005s
I20240502 21:26:37.832062 6048 consensus_queue.cc:260] T 2285212d78b0460ca329949eb3140e69 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:37.832221 6047 ts_tablet_manager.cc:1367] T a33fd7d5a8db41a39ba780e16a84f9b1 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20240502 21:26:37.832123 6046 raft_consensus.cc:348] T 9a3c115189e8442ba7c3fa7f65089d11 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:37.832329 6046 raft_consensus.cc:374] T 9a3c115189e8442ba7c3fa7f65089d11 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:37.832788 6048 ts_tablet_manager.cc:1398] T 2285212d78b0460ca329949eb3140e69 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:37.832942 6047 raft_consensus.cc:348] T a33fd7d5a8db41a39ba780e16a84f9b1 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:37.833122 6048 tablet_bootstrap.cc:492] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:37.833081 6045 raft_consensus.cc:348] T 9a3c115189e8442ba7c3fa7f65089d11 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:37.833192 6047 raft_consensus.cc:374] T a33fd7d5a8db41a39ba780e16a84f9b1 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:37.833273 6047 raft_consensus.cc:729] T a33fd7d5a8db41a39ba780e16a84f9b1 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:37.833323 6045 raft_consensus.cc:374] T 9a3c115189e8442ba7c3fa7f65089d11 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:37.833489 6045 raft_consensus.cc:729] T 9a3c115189e8442ba7c3fa7f65089d11 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:37.833539 6047 consensus_queue.cc:260] T a33fd7d5a8db41a39ba780e16a84f9b1 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:37.834123 6047 ts_tablet_manager.cc:1398] T a33fd7d5a8db41a39ba780e16a84f9b1 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:37.834558 6047 tablet_bootstrap.cc:492] T 9a3c115189e8442ba7c3fa7f65089d11 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:37.834859 6048 tablet_bootstrap.cc:654] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:37.836208 6047 tablet_bootstrap.cc:654] T 9a3c115189e8442ba7c3fa7f65089d11 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:37.837916 6046 raft_consensus.cc:729] T 9a3c115189e8442ba7c3fa7f65089d11 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:37.838382 6046 consensus_queue.cc:260] T 9a3c115189e8442ba7c3fa7f65089d11 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:37.839438 6046 ts_tablet_manager.cc:1398] T 9a3c115189e8442ba7c3fa7f65089d11 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 21:26:37.839910 6046 tablet_bootstrap.cc:492] T a33fd7d5a8db41a39ba780e16a84f9b1 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:37.842232 6046 tablet_bootstrap.cc:654] T a33fd7d5a8db41a39ba780e16a84f9b1 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:37.846907 6045 consensus_queue.cc:260] T 9a3c115189e8442ba7c3fa7f65089d11 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:37.847397 6047 tablet_bootstrap.cc:492] T 9a3c115189e8442ba7c3fa7f65089d11 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:37.848124 6045 ts_tablet_manager.cc:1398] T 9a3c115189e8442ba7c3fa7f65089d11 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240502 21:26:37.848484 6047 ts_tablet_manager.cc:1367] T 9a3c115189e8442ba7c3fa7f65089d11 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20240502 21:26:37.849221 6046 tablet_bootstrap.cc:492] T a33fd7d5a8db41a39ba780e16a84f9b1 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:37.849431 6046 ts_tablet_manager.cc:1367] T a33fd7d5a8db41a39ba780e16a84f9b1 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240502 21:26:37.849680 6048 tablet_bootstrap.cc:492] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:37.849480 6047 raft_consensus.cc:348] T 9a3c115189e8442ba7c3fa7f65089d11 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:37.849913 6048 ts_tablet_manager.cc:1367] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.008s
I20240502 21:26:37.849917 6047 raft_consensus.cc:374] T 9a3c115189e8442ba7c3fa7f65089d11 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:37.850293 6046 raft_consensus.cc:348] T a33fd7d5a8db41a39ba780e16a84f9b1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:37.850553 6046 raft_consensus.cc:374] T a33fd7d5a8db41a39ba780e16a84f9b1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:37.850648 6046 raft_consensus.cc:729] T a33fd7d5a8db41a39ba780e16a84f9b1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:37.850023 6047 raft_consensus.cc:729] T 9a3c115189e8442ba7c3fa7f65089d11 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:37.850895 6048 raft_consensus.cc:348] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:37.850953 6046 consensus_queue.cc:260] T a33fd7d5a8db41a39ba780e16a84f9b1 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:37.851150 6048 raft_consensus.cc:374] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:37.851081 6047 consensus_queue.cc:260] T 9a3c115189e8442ba7c3fa7f65089d11 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:37.851263 6048 raft_consensus.cc:729] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:37.851589 6048 consensus_queue.cc:260] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:37.852299 6048 ts_tablet_manager.cc:1398] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:37.852771 6046 ts_tablet_manager.cc:1398] T a33fd7d5a8db41a39ba780e16a84f9b1 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:37.853374 6047 ts_tablet_manager.cc:1398] T 9a3c115189e8442ba7c3fa7f65089d11 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:26:37.863881 5822 raft_consensus.cc:482] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:37.864024 5822 raft_consensus.cc:504] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:37.864661 5822 leader_election.cc:288] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:37.865113 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2285212d78b0460ca329949eb3140e69" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:37.865224 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2285212d78b0460ca329949eb3140e69" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:37.865345 4575 raft_consensus.cc:2454] T 2285212d78b0460ca329949eb3140e69 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:26:37.865598 4443 raft_consensus.cc:2454] T 2285212d78b0460ca329949eb3140e69 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:26:37.865778 4642 leader_election.cc:302] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:37.866075 5822 raft_consensus.cc:2781] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:37.866212 5822 raft_consensus.cc:482] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:37.866318 5822 raft_consensus.cc:3037] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:37.868304 5822 raft_consensus.cc:504] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:37.868824 5822 leader_election.cc:288] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Requested vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:37.869208 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2285212d78b0460ca329949eb3140e69" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:26:37.869271 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2285212d78b0460ca329949eb3140e69" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:37.869375 4575 raft_consensus.cc:3037] T 2285212d78b0460ca329949eb3140e69 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:37.869491 4442 raft_consensus.cc:3037] T 2285212d78b0460ca329949eb3140e69 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:37.871291 4575 raft_consensus.cc:2454] T 2285212d78b0460ca329949eb3140e69 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:26:37.871515 4442 raft_consensus.cc:2454] T 2285212d78b0460ca329949eb3140e69 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:26:37.871709 4642 leader_election.cc:302] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:37.871999 5822 raft_consensus.cc:2781] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:37.872383 5822 raft_consensus.cc:686] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Becoming Leader. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Running, Role: LEADER
I20240502 21:26:37.872673 5822 consensus_queue.cc:237] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:37.875600 4295 catalog_manager.cc:5484] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 398c3c8f3195492c900439ad5e1931bb (127.29.82.67). New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:26:37.930095 5938 raft_consensus.cc:482] T 9a3c115189e8442ba7c3fa7f65089d11 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:37.930269 5938 raft_consensus.cc:504] T 9a3c115189e8442ba7c3fa7f65089d11 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:37.930932 5938 leader_election.cc:288] T 9a3c115189e8442ba7c3fa7f65089d11 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:37.931254 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a3c115189e8442ba7c3fa7f65089d11" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:37.931460 4705 raft_consensus.cc:2454] T 9a3c115189e8442ba7c3fa7f65089d11 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:37.931468 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a3c115189e8442ba7c3fa7f65089d11" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:37.931757 4442 raft_consensus.cc:2454] T 9a3c115189e8442ba7c3fa7f65089d11 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:37.931885 4773 leader_election.cc:302] T 9a3c115189e8442ba7c3fa7f65089d11 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:37.932176 5938 raft_consensus.cc:2781] T 9a3c115189e8442ba7c3fa7f65089d11 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:37.932277 5938 raft_consensus.cc:482] T 9a3c115189e8442ba7c3fa7f65089d11 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:37.932372 5938 raft_consensus.cc:3037] T 9a3c115189e8442ba7c3fa7f65089d11 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:37.934280 5938 raft_consensus.cc:504] T 9a3c115189e8442ba7c3fa7f65089d11 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:37.934773 5938 leader_election.cc:288] T 9a3c115189e8442ba7c3fa7f65089d11 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:37.935307 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a3c115189e8442ba7c3fa7f65089d11" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:37.935497 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a3c115189e8442ba7c3fa7f65089d11" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:37.935561 4442 raft_consensus.cc:3037] T 9a3c115189e8442ba7c3fa7f65089d11 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:37.935725 4705 raft_consensus.cc:3037] T 9a3c115189e8442ba7c3fa7f65089d11 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:37.937785 4705 raft_consensus.cc:2454] T 9a3c115189e8442ba7c3fa7f65089d11 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:37.937809 4442 raft_consensus.cc:2454] T 9a3c115189e8442ba7c3fa7f65089d11 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:37.938213 4773 leader_election.cc:302] T 9a3c115189e8442ba7c3fa7f65089d11 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:37.938647 6005 raft_consensus.cc:2781] T 9a3c115189e8442ba7c3fa7f65089d11 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:37.939587 6005 raft_consensus.cc:686] T 9a3c115189e8442ba7c3fa7f65089d11 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:26:37.939947 6005 consensus_queue.cc:237] T 9a3c115189e8442ba7c3fa7f65089d11 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:37.942766 4295 catalog_manager.cc:5484] T 9a3c115189e8442ba7c3fa7f65089d11 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:26:38.040175 6024 raft_consensus.cc:482] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:38.040354 6024 raft_consensus.cc:504] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:38.040997 6024 leader_election.cc:288] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:38.041494 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a33fd7d5a8db41a39ba780e16a84f9b1" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:26:38.041457 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a33fd7d5a8db41a39ba780e16a84f9b1" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:38.041731 4838 raft_consensus.cc:2454] T a33fd7d5a8db41a39ba780e16a84f9b1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:26:38.041739 4705 raft_consensus.cc:2454] T a33fd7d5a8db41a39ba780e16a84f9b1 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:26:38.042132 4511 leader_election.cc:302] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:38.042404 6024 raft_consensus.cc:2781] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:38.042512 6024 raft_consensus.cc:482] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:38.042603 6024 raft_consensus.cc:3037] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:38.044564 6024 raft_consensus.cc:504] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:38.045076 6024 leader_election.cc:288] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:38.045526 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a33fd7d5a8db41a39ba780e16a84f9b1" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:26:38.045562 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a33fd7d5a8db41a39ba780e16a84f9b1" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:38.045746 4838 raft_consensus.cc:3037] T a33fd7d5a8db41a39ba780e16a84f9b1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:38.045759 4705 raft_consensus.cc:3037] T a33fd7d5a8db41a39ba780e16a84f9b1 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:38.048616 4705 raft_consensus.cc:2454] T a33fd7d5a8db41a39ba780e16a84f9b1 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:26:38.048790 4838 raft_consensus.cc:2454] T a33fd7d5a8db41a39ba780e16a84f9b1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:26:38.049082 4511 leader_election.cc:302] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:38.049371 6024 raft_consensus.cc:2781] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:38.049530 6024 raft_consensus.cc:686] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:26:38.049818 6024 consensus_queue.cc:237] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:38.052438 4295 catalog_manager.cc:5484] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:26:38.290668 5822 consensus_queue.cc:1035] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:38.304924 6058 consensus_queue.cc:1035] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:38.424031 6063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:38.424389 6063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:38.431934 6063 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:26:38.438913 6060 consensus_queue.cc:1035] T 9a3c115189e8442ba7c3fa7f65089d11 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:38.441763 6024 consensus_queue.cc:1035] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:38.447342 6023 consensus_queue.cc:1035] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:38.450181 6005 consensus_queue.cc:1035] T 9a3c115189e8442ba7c3fa7f65089d11 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_7'
INSERT report
rows total: 2000
time total: 203.182 ms
time per row: 0.101591 ms
I20240502 21:26:38.800979 4298 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:38.810323 4423 tablet_service.cc:1467] Processing CreateTablet for tablet 8f9c10843d63429fa2643e12050b9f29 (DEFAULT_TABLE table=rebalance_test_table_8 [id=e9b2eb99277d4f6a97dca65e69292ffd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:38.810346 4684 tablet_service.cc:1467] Processing CreateTablet for tablet 8f9c10843d63429fa2643e12050b9f29 (DEFAULT_TABLE table=rebalance_test_table_8 [id=e9b2eb99277d4f6a97dca65e69292ffd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:38.810801 4423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f9c10843d63429fa2643e12050b9f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:38.810819 4684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f9c10843d63429fa2643e12050b9f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:38.811937 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 350666c5e39744ff91d5adc8a6f2977d (DEFAULT_TABLE table=rebalance_test_table_8 [id=e9b2eb99277d4f6a97dca65e69292ffd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:38.812383 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 350666c5e39744ff91d5adc8a6f2977d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:38.814926 4555 tablet_service.cc:1467] Processing CreateTablet for tablet 8f9c10843d63429fa2643e12050b9f29 (DEFAULT_TABLE table=rebalance_test_table_8 [id=e9b2eb99277d4f6a97dca65e69292ffd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:38.815368 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f9c10843d63429fa2643e12050b9f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:38.820044 4554 tablet_service.cc:1467] Processing CreateTablet for tablet dad7605b83064a3abebd26b3b6570f27 (DEFAULT_TABLE table=rebalance_test_table_8 [id=e9b2eb99277d4f6a97dca65e69292ffd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:38.820550 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad7605b83064a3abebd26b3b6570f27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:38.824414 6126 tablet_bootstrap.cc:492] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:38.825672 4422 tablet_service.cc:1467] Processing CreateTablet for tablet 350666c5e39744ff91d5adc8a6f2977d (DEFAULT_TABLE table=rebalance_test_table_8 [id=e9b2eb99277d4f6a97dca65e69292ffd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:38.826077 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 350666c5e39744ff91d5adc8a6f2977d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:38.826367 6126 tablet_bootstrap.cc:654] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:38.826429 4818 tablet_service.cc:1467] Processing CreateTablet for tablet dad7605b83064a3abebd26b3b6570f27 (DEFAULT_TABLE table=rebalance_test_table_8 [id=e9b2eb99277d4f6a97dca65e69292ffd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:38.826965 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad7605b83064a3abebd26b3b6570f27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:38.831802 4817 tablet_service.cc:1467] Processing CreateTablet for tablet 350666c5e39744ff91d5adc8a6f2977d (DEFAULT_TABLE table=rebalance_test_table_8 [id=e9b2eb99277d4f6a97dca65e69292ffd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:38.832329 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 350666c5e39744ff91d5adc8a6f2977d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:38.835940 4423 tablet_service.cc:1467] Processing CreateTablet for tablet dad7605b83064a3abebd26b3b6570f27 (DEFAULT_TABLE table=rebalance_test_table_8 [id=e9b2eb99277d4f6a97dca65e69292ffd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:38.836973 4423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad7605b83064a3abebd26b3b6570f27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:38.852113 6128 tablet_bootstrap.cc:492] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:38.854984 6128 tablet_bootstrap.cc:654] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:38.856484 6129 tablet_bootstrap.cc:492] T 8f9c10843d63429fa2643e12050b9f29 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:38.858495 6129 tablet_bootstrap.cc:654] T 8f9c10843d63429fa2643e12050b9f29 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:38.861754 6125 tablet_bootstrap.cc:492] T 8f9c10843d63429fa2643e12050b9f29 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:38.863101 6129 tablet_bootstrap.cc:492] T 8f9c10843d63429fa2643e12050b9f29 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:38.863170 6126 tablet_bootstrap.cc:492] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:38.863345 6129 ts_tablet_manager.cc:1367] T 8f9c10843d63429fa2643e12050b9f29 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 21:26:38.863391 6126 ts_tablet_manager.cc:1367] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.039s user 0.018s sys 0.019s
I20240502 21:26:38.863600 6128 tablet_bootstrap.cc:492] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:38.863780 6128 ts_tablet_manager.cc:1367] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20240502 21:26:38.864387 6125 tablet_bootstrap.cc:654] T 8f9c10843d63429fa2643e12050b9f29 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:38.864327 6126 raft_consensus.cc:348] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:38.864398 6129 raft_consensus.cc:348] T 8f9c10843d63429fa2643e12050b9f29 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:38.864584 6129 raft_consensus.cc:374] T 8f9c10843d63429fa2643e12050b9f29 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:38.864615 6126 raft_consensus.cc:374] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:38.864606 6128 raft_consensus.cc:348] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:38.864722 6129 raft_consensus.cc:729] T 8f9c10843d63429fa2643e12050b9f29 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:38.864755 6126 raft_consensus.cc:729] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:38.864872 6128 raft_consensus.cc:374] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:38.865003 6128 raft_consensus.cc:729] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:38.865027 6129 consensus_queue.cc:260] T 8f9c10843d63429fa2643e12050b9f29 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:38.865054 6126 consensus_queue.cc:260] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:38.865943 6126 ts_tablet_manager.cc:1398] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:38.865274 6128 consensus_queue.cc:260] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:38.866298 6126 tablet_bootstrap.cc:492] T 350666c5e39744ff91d5adc8a6f2977d P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:38.866884 6128 ts_tablet_manager.cc:1398] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:26:38.867257 6128 tablet_bootstrap.cc:492] T dad7605b83064a3abebd26b3b6570f27 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:38.868187 6126 tablet_bootstrap.cc:654] T 350666c5e39744ff91d5adc8a6f2977d P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:38.868999 6128 tablet_bootstrap.cc:654] T dad7605b83064a3abebd26b3b6570f27 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:38.875845 6125 tablet_bootstrap.cc:492] T 8f9c10843d63429fa2643e12050b9f29 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:38.876099 6125 ts_tablet_manager.cc:1367] T 8f9c10843d63429fa2643e12050b9f29 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20240502 21:26:38.878123 6125 raft_consensus.cc:348] T 8f9c10843d63429fa2643e12050b9f29 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:38.878463 6125 raft_consensus.cc:374] T 8f9c10843d63429fa2643e12050b9f29 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:38.878613 6125 raft_consensus.cc:729] T 8f9c10843d63429fa2643e12050b9f29 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:38.878626 6129 ts_tablet_manager.cc:1398] T 8f9c10843d63429fa2643e12050b9f29 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20240502 21:26:38.879035 6129 tablet_bootstrap.cc:492] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:38.878942 6125 consensus_queue.cc:260] T 8f9c10843d63429fa2643e12050b9f29 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:38.879693 6125 ts_tablet_manager.cc:1398] T 8f9c10843d63429fa2643e12050b9f29 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:38.880056 6125 tablet_bootstrap.cc:492] T 350666c5e39744ff91d5adc8a6f2977d P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:38.881963 6125 tablet_bootstrap.cc:654] T 350666c5e39744ff91d5adc8a6f2977d P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:38.882915 6129 tablet_bootstrap.cc:654] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:38.886507 6129 tablet_bootstrap.cc:492] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:38.886822 6129 ts_tablet_manager.cc:1367] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20240502 21:26:38.887962 6129 raft_consensus.cc:348] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:38.888273 6129 raft_consensus.cc:374] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:38.888404 6129 raft_consensus.cc:729] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:38.888799 6129 consensus_queue.cc:260] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:38.891113 6129 ts_tablet_manager.cc:1398] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:26:38.892910 6126 tablet_bootstrap.cc:492] T 350666c5e39744ff91d5adc8a6f2977d P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:38.893121 6126 ts_tablet_manager.cc:1367] T 350666c5e39744ff91d5adc8a6f2977d P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.013s
I20240502 21:26:38.893174 6128 tablet_bootstrap.cc:492] T dad7605b83064a3abebd26b3b6570f27 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:38.893358 6128 ts_tablet_manager.cc:1367] T dad7605b83064a3abebd26b3b6570f27 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.026s user 0.013s sys 0.005s
I20240502 21:26:38.893965 6126 raft_consensus.cc:348] T 350666c5e39744ff91d5adc8a6f2977d P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:38.894120 6125 tablet_bootstrap.cc:492] T 350666c5e39744ff91d5adc8a6f2977d P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:38.894258 6126 raft_consensus.cc:374] T 350666c5e39744ff91d5adc8a6f2977d P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:38.894309 6125 ts_tablet_manager.cc:1367] T 350666c5e39744ff91d5adc8a6f2977d P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20240502 21:26:38.894372 6126 raft_consensus.cc:729] T 350666c5e39744ff91d5adc8a6f2977d P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:38.894246 6128 raft_consensus.cc:348] T dad7605b83064a3abebd26b3b6570f27 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:38.894615 6128 raft_consensus.cc:374] T dad7605b83064a3abebd26b3b6570f27 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:38.894743 6128 raft_consensus.cc:729] T dad7605b83064a3abebd26b3b6570f27 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:38.895047 6128 consensus_queue.cc:260] T dad7605b83064a3abebd26b3b6570f27 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:38.895265 6125 raft_consensus.cc:348] T 350666c5e39744ff91d5adc8a6f2977d P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:38.895609 6125 raft_consensus.cc:374] T 350666c5e39744ff91d5adc8a6f2977d P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:38.895748 6125 raft_consensus.cc:729] T 350666c5e39744ff91d5adc8a6f2977d P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:38.895639 6126 consensus_queue.cc:260] T 350666c5e39744ff91d5adc8a6f2977d P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:38.896168 6125 consensus_queue.cc:260] T 350666c5e39744ff91d5adc8a6f2977d P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:38.896412 6126 ts_tablet_manager.cc:1398] T 350666c5e39744ff91d5adc8a6f2977d P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:38.896802 6126 tablet_bootstrap.cc:492] T dad7605b83064a3abebd26b3b6570f27 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:38.898535 6126 tablet_bootstrap.cc:654] T dad7605b83064a3abebd26b3b6570f27 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:38.898983 6128 ts_tablet_manager.cc:1398] T dad7605b83064a3abebd26b3b6570f27 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 21:26:38.899833 6125 ts_tablet_manager.cc:1398] T 350666c5e39744ff91d5adc8a6f2977d P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:26:38.903205 6126 tablet_bootstrap.cc:492] T dad7605b83064a3abebd26b3b6570f27 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:38.903364 6126 ts_tablet_manager.cc:1367] T dad7605b83064a3abebd26b3b6570f27 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 21:26:38.904042 6126 raft_consensus.cc:348] T dad7605b83064a3abebd26b3b6570f27 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:38.904199 6126 raft_consensus.cc:374] T dad7605b83064a3abebd26b3b6570f27 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:38.904266 6126 raft_consensus.cc:729] T dad7605b83064a3abebd26b3b6570f27 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:38.904430 6126 consensus_queue.cc:260] T dad7605b83064a3abebd26b3b6570f27 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:38.904896 6126 ts_tablet_manager.cc:1398] T dad7605b83064a3abebd26b3b6570f27 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 21:26:38.946600 6024 raft_consensus.cc:482] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:38.946770 6024 raft_consensus.cc:504] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:38.947353 6024 leader_election.cc:288] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:38.947808 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad7605b83064a3abebd26b3b6570f27" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:26:38.947887 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad7605b83064a3abebd26b3b6570f27" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:38.948056 4838 raft_consensus.cc:2454] T dad7605b83064a3abebd26b3b6570f27 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:26:38.948101 4442 raft_consensus.cc:2454] T dad7605b83064a3abebd26b3b6570f27 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:26:38.948465 4508 leader_election.cc:302] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:38.948757 6024 raft_consensus.cc:2781] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:38.948863 6024 raft_consensus.cc:482] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:38.948956 6024 raft_consensus.cc:3037] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:38.950865 6024 raft_consensus.cc:504] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:38.951388 6024 leader_election.cc:288] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Requested vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:38.951820 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad7605b83064a3abebd26b3b6570f27" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:26:38.951974 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad7605b83064a3abebd26b3b6570f27" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:38.952035 4838 raft_consensus.cc:3037] T dad7605b83064a3abebd26b3b6570f27 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:38.952172 4442 raft_consensus.cc:3037] T dad7605b83064a3abebd26b3b6570f27 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:38.954033 4838 raft_consensus.cc:2454] T dad7605b83064a3abebd26b3b6570f27 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:26:38.954084 4442 raft_consensus.cc:2454] T dad7605b83064a3abebd26b3b6570f27 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:26:38.954394 4508 leader_election.cc:302] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:38.954645 6024 raft_consensus.cc:2781] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:38.954772 6024 raft_consensus.cc:686] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:26:38.955037 6024 consensus_queue.cc:237] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:38.957791 4298 catalog_manager.cc:5484] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:26:39.069571 6077 raft_consensus.cc:482] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:39.069753 6077 raft_consensus.cc:504] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:39.070344 6077 leader_election.cc:288] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:39.070864 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350666c5e39744ff91d5adc8a6f2977d" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:39.070955 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350666c5e39744ff91d5adc8a6f2977d" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:39.071116 4442 raft_consensus.cc:2454] T 350666c5e39744ff91d5adc8a6f2977d P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:39.071204 4706 raft_consensus.cc:2454] T 350666c5e39744ff91d5adc8a6f2977d P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:39.071511 4770 leader_election.cc:302] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:39.071810 6077 raft_consensus.cc:2781] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:39.071906 6077 raft_consensus.cc:482] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:39.071998 6077 raft_consensus.cc:3037] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:39.073902 6077 raft_consensus.cc:504] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:39.074419 6077 leader_election.cc:288] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:39.074856 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350666c5e39744ff91d5adc8a6f2977d" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:39.075047 4442 raft_consensus.cc:3037] T 350666c5e39744ff91d5adc8a6f2977d P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:39.074962 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350666c5e39744ff91d5adc8a6f2977d" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:39.075193 4706 raft_consensus.cc:3037] T 350666c5e39744ff91d5adc8a6f2977d P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:39.077183 4706 raft_consensus.cc:2454] T 350666c5e39744ff91d5adc8a6f2977d P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:39.077252 4442 raft_consensus.cc:2454] T 350666c5e39744ff91d5adc8a6f2977d P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:39.077606 4773 leader_election.cc:302] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:39.077927 6077 raft_consensus.cc:2781] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:39.078065 6077 raft_consensus.cc:686] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:26:39.078315 6077 consensus_queue.cc:237] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:39.081344 4298 catalog_manager.cc:5484] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:26:39.089287 5723 raft_consensus.cc:482] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:39.089457 5723 raft_consensus.cc:504] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:39.090058 5723 leader_election.cc:288] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:39.090472 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f9c10843d63429fa2643e12050b9f29" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:39.090559 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f9c10843d63429fa2643e12050b9f29" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:39.090730 4706 raft_consensus.cc:2454] T 8f9c10843d63429fa2643e12050b9f29 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:26:39.090855 4575 raft_consensus.cc:2454] T 8f9c10843d63429fa2643e12050b9f29 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:26:39.091094 4378 leader_election.cc:302] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:26:39.091369 5723 raft_consensus.cc:2781] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:39.091471 5723 raft_consensus.cc:482] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:39.091604 5723 raft_consensus.cc:3037] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:39.093602 5723 raft_consensus.cc:504] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:39.094125 5723 leader_election.cc:288] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:39.094568 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f9c10843d63429fa2643e12050b9f29" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:39.094573 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f9c10843d63429fa2643e12050b9f29" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:26:39.094751 4706 raft_consensus.cc:3037] T 8f9c10843d63429fa2643e12050b9f29 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:39.094764 4575 raft_consensus.cc:3037] T 8f9c10843d63429fa2643e12050b9f29 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:39.096627 4706 raft_consensus.cc:2454] T 8f9c10843d63429fa2643e12050b9f29 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:26:39.096704 4575 raft_consensus.cc:2454] T 8f9c10843d63429fa2643e12050b9f29 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:26:39.097019 4378 leader_election.cc:302] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:26:39.097257 5723 raft_consensus.cc:2781] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:39.097419 5723 raft_consensus.cc:686] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:26:39.097671 5723 consensus_queue.cc:237] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:39.100615 4298 catalog_manager.cc:5484] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:26:39.368088 6024 consensus_queue.cc:1035] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:39.373384 6024 consensus_queue.cc:1035] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:39.489646 6142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:39.489964 6142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:39.497740 6142 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:26:39.498906 6143 consensus_queue.cc:1035] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:39.503996 6143 consensus_queue.cc:1035] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_8'
I20240502 21:26:39.546176 6077 consensus_queue.cc:1035] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:39.551399 6077 consensus_queue.cc:1035] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 259.014 ms
time per row: 0.129507 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:26:40.863736 4298 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:40.873383 4555 tablet_service.cc:1467] Processing CreateTablet for tablet c9fe2e18aa6c407fbbcbb74bf45bd520 (DEFAULT_TABLE table=rebalance_test_table_9 [id=80b493b0fc3f49dcbf948cee9e748e3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:40.873615 4818 tablet_service.cc:1467] Processing CreateTablet for tablet c9fe2e18aa6c407fbbcbb74bf45bd520 (DEFAULT_TABLE table=rebalance_test_table_9 [id=80b493b0fc3f49dcbf948cee9e748e3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:40.873871 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9fe2e18aa6c407fbbcbb74bf45bd520. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:40.874466 4817 tablet_service.cc:1467] Processing CreateTablet for tablet 5f614ef09d764740926c10618370084f (DEFAULT_TABLE table=rebalance_test_table_9 [id=80b493b0fc3f49dcbf948cee9e748e3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:40.874927 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9fe2e18aa6c407fbbcbb74bf45bd520. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:40.875876 4423 tablet_service.cc:1467] Processing CreateTablet for tablet a607fad427764c46b56f8963952d7613 (DEFAULT_TABLE table=rebalance_test_table_9 [id=80b493b0fc3f49dcbf948cee9e748e3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:40.876336 4423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a607fad427764c46b56f8963952d7613. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:40.876402 4684 tablet_service.cc:1467] Processing CreateTablet for tablet c9fe2e18aa6c407fbbcbb74bf45bd520 (DEFAULT_TABLE table=rebalance_test_table_9 [id=80b493b0fc3f49dcbf948cee9e748e3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:40.876814 4684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9fe2e18aa6c407fbbcbb74bf45bd520. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:40.879896 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f614ef09d764740926c10618370084f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:40.880087 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 5f614ef09d764740926c10618370084f (DEFAULT_TABLE table=rebalance_test_table_9 [id=80b493b0fc3f49dcbf948cee9e748e3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:40.880578 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f614ef09d764740926c10618370084f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:40.881182 4554 tablet_service.cc:1467] Processing CreateTablet for tablet 5f614ef09d764740926c10618370084f (DEFAULT_TABLE table=rebalance_test_table_9 [id=80b493b0fc3f49dcbf948cee9e748e3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:40.881561 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f614ef09d764740926c10618370084f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:40.887295 4685 tablet_service.cc:1467] Processing CreateTablet for tablet a607fad427764c46b56f8963952d7613 (DEFAULT_TABLE table=rebalance_test_table_9 [id=80b493b0fc3f49dcbf948cee9e748e3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:40.887748 4685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a607fad427764c46b56f8963952d7613. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:40.888070 4553 tablet_service.cc:1467] Processing CreateTablet for tablet a607fad427764c46b56f8963952d7613 (DEFAULT_TABLE table=rebalance_test_table_9 [id=80b493b0fc3f49dcbf948cee9e748e3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:40.888501 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a607fad427764c46b56f8963952d7613. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:40.888626 6209 tablet_bootstrap.cc:492] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:40.890415 6209 tablet_bootstrap.cc:654] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:40.894327 6212 tablet_bootstrap.cc:492] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:40.897840 6212 tablet_bootstrap.cc:654] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:40.902040 6209 tablet_bootstrap.cc:492] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:40.902302 6209 ts_tablet_manager.cc:1367] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.009s
I20240502 21:26:40.902570 6211 tablet_bootstrap.cc:492] T a607fad427764c46b56f8963952d7613 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:40.902839 6213 tablet_bootstrap.cc:492] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:40.903169 6209 raft_consensus.cc:348] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:40.903503 6209 raft_consensus.cc:374] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:40.903693 6209 raft_consensus.cc:729] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:40.903960 6209 consensus_queue.cc:260] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:40.904445 6209 ts_tablet_manager.cc:1398] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:40.904457 6211 tablet_bootstrap.cc:654] T a607fad427764c46b56f8963952d7613 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:40.905079 6213 tablet_bootstrap.cc:654] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:40.907007 6209 tablet_bootstrap.cc:492] T 5f614ef09d764740926c10618370084f P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:40.907711 6213 tablet_bootstrap.cc:492] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:40.907956 6213 ts_tablet_manager.cc:1367] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 21:26:40.908668 6209 tablet_bootstrap.cc:654] T 5f614ef09d764740926c10618370084f P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:40.909065 6213 raft_consensus.cc:348] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:40.909421 6213 raft_consensus.cc:374] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:40.909610 6213 raft_consensus.cc:729] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:40.910148 6212 tablet_bootstrap.cc:492] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:40.910045 6213 consensus_queue.cc:260] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:40.910316 6212 ts_tablet_manager.cc:1367] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240502 21:26:40.911015 6213 ts_tablet_manager.cc:1398] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:40.911648 6213 tablet_bootstrap.cc:492] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:40.911733 6212 raft_consensus.cc:348] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:40.912015 6212 raft_consensus.cc:374] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:40.912168 6212 raft_consensus.cc:729] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:40.912402 6212 consensus_queue.cc:260] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:40.913251 6213 tablet_bootstrap.cc:654] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:40.913854 6212 ts_tablet_manager.cc:1398] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:40.914191 6212 tablet_bootstrap.cc:492] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:40.915836 6212 tablet_bootstrap.cc:654] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:40.917623 6209 tablet_bootstrap.cc:492] T 5f614ef09d764740926c10618370084f P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:40.917851 6209 ts_tablet_manager.cc:1367] T 5f614ef09d764740926c10618370084f P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240502 21:26:40.918691 6209 raft_consensus.cc:348] T 5f614ef09d764740926c10618370084f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:40.918992 6209 raft_consensus.cc:374] T 5f614ef09d764740926c10618370084f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:40.919049 6213 tablet_bootstrap.cc:492] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:40.919131 6209 raft_consensus.cc:729] T 5f614ef09d764740926c10618370084f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:40.919337 6213 ts_tablet_manager.cc:1367] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 21:26:40.919493 6209 consensus_queue.cc:260] T 5f614ef09d764740926c10618370084f P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:40.919799 6117 raft_consensus.cc:482] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:40.919937 6117 raft_consensus.cc:504] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:40.920353 6213 raft_consensus.cc:348] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:40.920629 6213 raft_consensus.cc:374] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:40.920711 6213 raft_consensus.cc:729] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:40.920696 6117 leader_election.cc:288] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:40.921005 6213 consensus_queue.cc:260] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:40.921835 6213 ts_tablet_manager.cc:1398] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:40.921952 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fe2e18aa6c407fbbcbb74bf45bd520" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:26:40.922200 4838 raft_consensus.cc:2454] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:26:40.922642 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fe2e18aa6c407fbbcbb74bf45bd520" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
W20240502 21:26:40.923614 4511 leader_election.cc:341] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:40.924139 4508 leader_election.cc:302] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters: 398c3c8f3195492c900439ad5e1931bb
I20240502 21:26:40.925047 6117 raft_consensus.cc:2781] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:40.925174 6117 raft_consensus.cc:482] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:40.925273 6117 raft_consensus.cc:3037] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:40.927855 6212 tablet_bootstrap.cc:492] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:40.928083 6212 ts_tablet_manager.cc:1367] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20240502 21:26:40.928323 6211 tablet_bootstrap.cc:492] T a607fad427764c46b56f8963952d7613 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:40.928543 6211 ts_tablet_manager.cc:1367] T a607fad427764c46b56f8963952d7613 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.026s user 0.017s sys 0.004s
I20240502 21:26:40.929075 6209 ts_tablet_manager.cc:1398] T 5f614ef09d764740926c10618370084f P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20240502 21:26:40.929426 6209 tablet_bootstrap.cc:492] T a607fad427764c46b56f8963952d7613 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:40.929445 6211 raft_consensus.cc:348] T a607fad427764c46b56f8963952d7613 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:40.929723 6211 raft_consensus.cc:374] T a607fad427764c46b56f8963952d7613 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:40.929843 6211 raft_consensus.cc:729] T a607fad427764c46b56f8963952d7613 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:40.930102 6211 consensus_queue.cc:260] T a607fad427764c46b56f8963952d7613 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:40.930814 6211 ts_tablet_manager.cc:1398] T a607fad427764c46b56f8963952d7613 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:40.931456 6117 raft_consensus.cc:504] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:40.932173 6209 tablet_bootstrap.cc:654] T a607fad427764c46b56f8963952d7613 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:40.933003 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fe2e18aa6c407fbbcbb74bf45bd520" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:26:40.933230 4838 raft_consensus.cc:3037] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:40.929019 6212 raft_consensus.cc:348] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:40.934293 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fe2e18aa6c407fbbcbb74bf45bd520" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:40.934610 6212 raft_consensus.cc:374] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:40.934898 6212 raft_consensus.cc:729] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
W20240502 21:26:40.934978 4511 leader_election.cc:341] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:40.935334 6212 consensus_queue.cc:260] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:40.936985 6212 ts_tablet_manager.cc:1398] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240502 21:26:40.937389 6212 tablet_bootstrap.cc:492] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:40.937410 4838 raft_consensus.cc:2454] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:26:40.938669 4508 leader_election.cc:302] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters: 398c3c8f3195492c900439ad5e1931bb
I20240502 21:26:40.938973 6219 raft_consensus.cc:2781] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:40.939514 6212 tablet_bootstrap.cc:654] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:40.940723 6117 leader_election.cc:288] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Requested vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:40.948377 6219 raft_consensus.cc:686] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:26:40.948813 6219 consensus_queue.cc:237] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:40.950191 6212 tablet_bootstrap.cc:492] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:40.950387 6212 ts_tablet_manager.cc:1367] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.006s
I20240502 21:26:40.951258 6209 tablet_bootstrap.cc:492] T a607fad427764c46b56f8963952d7613 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:40.951241 6212 raft_consensus.cc:348] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:40.951452 6212 raft_consensus.cc:374] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:40.951472 6209 ts_tablet_manager.cc:1367] T a607fad427764c46b56f8963952d7613 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.000s
I20240502 21:26:40.951551 6212 raft_consensus.cc:729] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:40.951853 6212 consensus_queue.cc:260] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:40.952322 6209 raft_consensus.cc:348] T a607fad427764c46b56f8963952d7613 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:40.952605 6209 raft_consensus.cc:374] T a607fad427764c46b56f8963952d7613 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:40.953225 6212 ts_tablet_manager.cc:1398] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240502 21:26:40.952998 4295 catalog_manager.cc:5484] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:26:40.953303 6209 raft_consensus.cc:729] T a607fad427764c46b56f8963952d7613 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:40.953557 6209 consensus_queue.cc:260] T a607fad427764c46b56f8963952d7613 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:40.954118 6209 ts_tablet_manager.cc:1398] T a607fad427764c46b56f8963952d7613 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:26:40.979156 6058 raft_consensus.cc:482] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:40.979341 6058 raft_consensus.cc:504] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:40.980021 6058 leader_election.cc:288] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:40.980504 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f614ef09d764740926c10618370084f" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:26:40.980584 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f614ef09d764740926c10618370084f" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:40.980752 4838 raft_consensus.cc:2454] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:26:40.980852 4575 raft_consensus.cc:2454] T 5f614ef09d764740926c10618370084f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:26:40.981243 4639 leader_election.cc:302] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:40.981521 6058 raft_consensus.cc:2781] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:40.981607 6058 raft_consensus.cc:482] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:40.981712 6058 raft_consensus.cc:3037] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:40.983629 6058 raft_consensus.cc:504] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:40.984198 6058 leader_election.cc:288] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Requested vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:40.984622 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f614ef09d764740926c10618370084f" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:26:40.984704 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f614ef09d764740926c10618370084f" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:26:40.984833 4838 raft_consensus.cc:3037] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:40.984944 4575 raft_consensus.cc:3037] T 5f614ef09d764740926c10618370084f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:40.987112 4838 raft_consensus.cc:2454] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:26:40.987627 4639 leader_election.cc:302] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:40.987843 4575 raft_consensus.cc:2454] T 5f614ef09d764740926c10618370084f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:26:40.987941 6058 raft_consensus.cc:2781] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:40.988387 6058 raft_consensus.cc:686] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Becoming Leader. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Running, Role: LEADER
I20240502 21:26:40.988720 6058 consensus_queue.cc:237] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:40.991861 4295 catalog_manager.cc:5484] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 398c3c8f3195492c900439ad5e1931bb (127.29.82.67). New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:26:41.092864 6058 raft_consensus.cc:482] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:41.093061 6058 raft_consensus.cc:504] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:41.093709 6058 leader_election.cc:288] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:41.094156 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a607fad427764c46b56f8963952d7613" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:41.094184 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a607fad427764c46b56f8963952d7613" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:41.094426 4575 raft_consensus.cc:2454] T a607fad427764c46b56f8963952d7613 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:26:41.094426 4442 raft_consensus.cc:2454] T a607fad427764c46b56f8963952d7613 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:26:41.094841 4642 leader_election.cc:302] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:41.095125 6058 raft_consensus.cc:2781] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:41.095263 6058 raft_consensus.cc:482] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:41.095366 6058 raft_consensus.cc:3037] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:41.097376 6058 raft_consensus.cc:504] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:41.097903 6058 leader_election.cc:288] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:41.098311 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a607fad427764c46b56f8963952d7613" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:41.098346 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a607fad427764c46b56f8963952d7613" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:26:41.098493 4442 raft_consensus.cc:3037] T a607fad427764c46b56f8963952d7613 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:41.098548 4575 raft_consensus.cc:3037] T a607fad427764c46b56f8963952d7613 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:41.100384 4575 raft_consensus.cc:2454] T a607fad427764c46b56f8963952d7613 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:26:41.100397 4442 raft_consensus.cc:2454] T a607fad427764c46b56f8963952d7613 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:26:41.100771 4639 leader_election.cc:302] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:26:41.101047 6058 raft_consensus.cc:2781] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:41.101193 6058 raft_consensus.cc:686] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Becoming Leader. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Running, Role: LEADER
I20240502 21:26:41.101466 6058 consensus_queue.cc:237] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:41.104588 4295 catalog_manager.cc:5484] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 398c3c8f3195492c900439ad5e1931bb (127.29.82.67). New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:26:41.353560 4705 raft_consensus.cc:3037] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:41.356304 6117 consensus_queue.cc:1035] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:41.361974 6117 consensus_queue.cc:1035] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:41.380334 6058 consensus_queue.cc:1035] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:41.386632 6226 consensus_queue.cc:1035] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:41.485261 6229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:41.485556 6229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:41.493103 6229 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:26:41.517513 6226 consensus_queue.cc:1035] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:41.521852 6226 consensus_queue.cc:1035] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
W20240502 21:26:41.710954 4642 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.66:35781, user_credentials={real_user=slave}} blocked reactor thread for 47175.5us
INSERT report
rows total: 2000
time total: 243.191 ms
time per row: 0.121595 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:26:42.863459 4291 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:42.873183 4423 tablet_service.cc:1467] Processing CreateTablet for tablet e733125cd70d474cb75da730cc73c2ef (DEFAULT_TABLE table=rebalance_test_table_10 [id=b0aca53acbb943f78aa824def28f5058]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:42.873378 4817 tablet_service.cc:1467] Processing CreateTablet for tablet e733125cd70d474cb75da730cc73c2ef (DEFAULT_TABLE table=rebalance_test_table_10 [id=b0aca53acbb943f78aa824def28f5058]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:42.873705 4423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e733125cd70d474cb75da730cc73c2ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:42.873876 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e733125cd70d474cb75da730cc73c2ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:42.874329 4684 tablet_service.cc:1467] Processing CreateTablet for tablet dfcd9d2ebaad42d2a2066a0f995f42a9 (DEFAULT_TABLE table=rebalance_test_table_10 [id=b0aca53acbb943f78aa824def28f5058]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:42.874727 4684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfcd9d2ebaad42d2a2066a0f995f42a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:42.878419 6295 tablet_bootstrap.cc:492] T e733125cd70d474cb75da730cc73c2ef P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:42.880025 6295 tablet_bootstrap.cc:654] T e733125cd70d474cb75da730cc73c2ef P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:42.880421 4553 tablet_service.cc:1467] Processing CreateTablet for tablet e733125cd70d474cb75da730cc73c2ef (DEFAULT_TABLE table=rebalance_test_table_10 [id=b0aca53acbb943f78aa824def28f5058]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:42.880982 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e733125cd70d474cb75da730cc73c2ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:42.884778 6295 tablet_bootstrap.cc:492] T e733125cd70d474cb75da730cc73c2ef P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:42.884781 4554 tablet_service.cc:1467] Processing CreateTablet for tablet 1c49766c76da48b1b35c8837e5816a71 (DEFAULT_TABLE table=rebalance_test_table_10 [id=b0aca53acbb943f78aa824def28f5058]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:42.885195 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c49766c76da48b1b35c8837e5816a71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:42.885305 4422 tablet_service.cc:1467] Processing CreateTablet for tablet dfcd9d2ebaad42d2a2066a0f995f42a9 (DEFAULT_TABLE table=rebalance_test_table_10 [id=b0aca53acbb943f78aa824def28f5058]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:42.885705 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfcd9d2ebaad42d2a2066a0f995f42a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:42.887310 4421 tablet_service.cc:1467] Processing CreateTablet for tablet 1c49766c76da48b1b35c8837e5816a71 (DEFAULT_TABLE table=rebalance_test_table_10 [id=b0aca53acbb943f78aa824def28f5058]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:42.887794 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c49766c76da48b1b35c8837e5816a71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:42.888465 4817 tablet_service.cc:1467] Processing CreateTablet for tablet dfcd9d2ebaad42d2a2066a0f995f42a9 (DEFAULT_TABLE table=rebalance_test_table_10 [id=b0aca53acbb943f78aa824def28f5058]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:42.888986 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfcd9d2ebaad42d2a2066a0f995f42a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:42.893919 4818 tablet_service.cc:1467] Processing CreateTablet for tablet 1c49766c76da48b1b35c8837e5816a71 (DEFAULT_TABLE table=rebalance_test_table_10 [id=b0aca53acbb943f78aa824def28f5058]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:42.894313 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c49766c76da48b1b35c8837e5816a71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:42.897969 6295 ts_tablet_manager.cc:1367] T e733125cd70d474cb75da730cc73c2ef P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.003s
I20240502 21:26:42.899185 6295 raft_consensus.cc:348] T e733125cd70d474cb75da730cc73c2ef P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:42.899488 6295 raft_consensus.cc:374] T e733125cd70d474cb75da730cc73c2ef P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:42.899670 6295 raft_consensus.cc:729] T e733125cd70d474cb75da730cc73c2ef P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:42.900032 6295 consensus_queue.cc:260] T e733125cd70d474cb75da730cc73c2ef P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:42.901037 6295 ts_tablet_manager.cc:1398] T e733125cd70d474cb75da730cc73c2ef P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:26:42.901345 6295 tablet_bootstrap.cc:492] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:42.902060 6299 tablet_bootstrap.cc:492] T 1c49766c76da48b1b35c8837e5816a71 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:42.903307 6295 tablet_bootstrap.cc:654] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:42.901360 6298 tablet_bootstrap.cc:492] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:42.904299 6299 tablet_bootstrap.cc:654] T 1c49766c76da48b1b35c8837e5816a71 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:42.905719 6298 tablet_bootstrap.cc:654] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:42.911062 6296 tablet_bootstrap.cc:492] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:42.911612 6298 tablet_bootstrap.cc:492] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:42.911837 6298 ts_tablet_manager.cc:1367] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240502 21:26:42.913107 6296 tablet_bootstrap.cc:654] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:42.913898 6298 raft_consensus.cc:348] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:42.914201 6298 raft_consensus.cc:374] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:42.914366 6298 raft_consensus.cc:729] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:42.914748 6298 consensus_queue.cc:260] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:42.915625 6298 ts_tablet_manager.cc:1398] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:26:42.917415 6295 tablet_bootstrap.cc:492] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:42.917639 6295 ts_tablet_manager.cc:1367] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.011s
I20240502 21:26:42.919195 6295 raft_consensus.cc:348] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:42.919660 6295 raft_consensus.cc:374] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:42.919739 6299 tablet_bootstrap.cc:492] T 1c49766c76da48b1b35c8837e5816a71 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:42.919806 6295 raft_consensus.cc:729] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:42.920114 6299 ts_tablet_manager.cc:1367] T 1c49766c76da48b1b35c8837e5816a71 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.012s
I20240502 21:26:42.920101 6295 consensus_queue.cc:260] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:42.920733 6295 ts_tablet_manager.cc:1398] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:26:42.921078 6295 tablet_bootstrap.cc:492] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:42.921371 6299 raft_consensus.cc:348] T 1c49766c76da48b1b35c8837e5816a71 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:42.921732 6299 raft_consensus.cc:374] T 1c49766c76da48b1b35c8837e5816a71 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:42.921896 6299 raft_consensus.cc:729] T 1c49766c76da48b1b35c8837e5816a71 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:42.922281 6299 consensus_queue.cc:260] T 1c49766c76da48b1b35c8837e5816a71 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:42.922721 6295 tablet_bootstrap.cc:654] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:42.923111 6299 ts_tablet_manager.cc:1398] T 1c49766c76da48b1b35c8837e5816a71 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:42.923416 6299 tablet_bootstrap.cc:492] T e733125cd70d474cb75da730cc73c2ef P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:42.923954 6296 tablet_bootstrap.cc:492] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:42.924160 6296 ts_tablet_manager.cc:1367] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20240502 21:26:42.925010 6296 raft_consensus.cc:348] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:42.925182 6299 tablet_bootstrap.cc:654] T e733125cd70d474cb75da730cc73c2ef P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:42.925307 6296 raft_consensus.cc:374] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:42.925457 6296 raft_consensus.cc:729] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:42.925778 6296 consensus_queue.cc:260] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:42.930490 6296 ts_tablet_manager.cc:1398] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 21:26:42.935883 6295 tablet_bootstrap.cc:492] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:42.936055 6295 ts_tablet_manager.cc:1367] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20240502 21:26:42.936904 6299 tablet_bootstrap.cc:492] T e733125cd70d474cb75da730cc73c2ef P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:42.937111 6299 ts_tablet_manager.cc:1367] T e733125cd70d474cb75da730cc73c2ef P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20240502 21:26:42.937251 6296 tablet_bootstrap.cc:492] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:42.938297 6299 raft_consensus.cc:348] T e733125cd70d474cb75da730cc73c2ef P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:42.938560 6299 raft_consensus.cc:374] T e733125cd70d474cb75da730cc73c2ef P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:42.938675 6299 raft_consensus.cc:729] T e733125cd70d474cb75da730cc73c2ef P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:42.938663 6295 raft_consensus.cc:348] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:42.938889 6295 raft_consensus.cc:374] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:42.939025 6295 raft_consensus.cc:729] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:42.939059 6296 tablet_bootstrap.cc:654] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:42.938959 6299 consensus_queue.cc:260] T e733125cd70d474cb75da730cc73c2ef P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:42.939327 6295 consensus_queue.cc:260] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:42.939702 6299 ts_tablet_manager.cc:1398] T e733125cd70d474cb75da730cc73c2ef P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:42.940188 6295 ts_tablet_manager.cc:1398] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 21:26:42.941627 6296 tablet_bootstrap.cc:492] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:42.941905 6296 ts_tablet_manager.cc:1367] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 21:26:42.942574 6296 raft_consensus.cc:348] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:42.942755 6296 raft_consensus.cc:374] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:42.942842 6296 raft_consensus.cc:729] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:42.943126 6296 consensus_queue.cc:260] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:42.944571 6296 ts_tablet_manager.cc:1398] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:42.944815 6296 tablet_bootstrap.cc:492] T 1c49766c76da48b1b35c8837e5816a71 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:42.945992 6296 tablet_bootstrap.cc:654] T 1c49766c76da48b1b35c8837e5816a71 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:42.947698 6296 tablet_bootstrap.cc:492] T 1c49766c76da48b1b35c8837e5816a71 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:42.947844 6296 ts_tablet_manager.cc:1367] T 1c49766c76da48b1b35c8837e5816a71 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:42.948604 6296 raft_consensus.cc:348] T 1c49766c76da48b1b35c8837e5816a71 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:42.948829 6296 raft_consensus.cc:374] T 1c49766c76da48b1b35c8837e5816a71 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:42.948930 6296 raft_consensus.cc:729] T 1c49766c76da48b1b35c8837e5816a71 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:42.949108 6296 consensus_queue.cc:260] T 1c49766c76da48b1b35c8837e5816a71 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:42.949573 6296 ts_tablet_manager.cc:1398] T 1c49766c76da48b1b35c8837e5816a71 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:26:42.964777 6058 raft_consensus.cc:482] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:42.964938 6058 raft_consensus.cc:504] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:42.965484 6058 leader_election.cc:288] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:42.965942 4441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfcd9d2ebaad42d2a2066a0f995f42a9" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:42.966027 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfcd9d2ebaad42d2a2066a0f995f42a9" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:26:42.966122 4441 raft_consensus.cc:2454] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:26:42.966264 4839 raft_consensus.cc:2454] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:26:42.966601 4639 leader_election.cc:302] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:26:42.966888 6058 raft_consensus.cc:2781] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:42.967028 6058 raft_consensus.cc:482] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:42.967132 6058 raft_consensus.cc:3037] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:42.969034 6058 raft_consensus.cc:504] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:42.969540 6058 leader_election.cc:288] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:42.969938 4441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfcd9d2ebaad42d2a2066a0f995f42a9" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:42.969991 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfcd9d2ebaad42d2a2066a0f995f42a9" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:26:42.970108 4441 raft_consensus.cc:3037] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:42.970145 4839 raft_consensus.cc:3037] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:42.972052 4441 raft_consensus.cc:2454] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:26:42.972062 4839 raft_consensus.cc:2454] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:26:42.972474 4639 leader_election.cc:302] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:42.972739 6058 raft_consensus.cc:2781] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:42.973093 6058 raft_consensus.cc:686] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Becoming Leader. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Running, Role: LEADER
I20240502 21:26:42.973346 6058 consensus_queue.cc:237] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:42.976423 4294 catalog_manager.cc:5484] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 398c3c8f3195492c900439ad5e1931bb (127.29.82.67). New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:26:43.077602 5723 raft_consensus.cc:482] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:43.077803 5723 raft_consensus.cc:504] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:43.078467 5723 leader_election.cc:288] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:43.078934 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e733125cd70d474cb75da730cc73c2ef" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:26:43.078987 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e733125cd70d474cb75da730cc73c2ef" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:43.079165 4839 raft_consensus.cc:2454] T e733125cd70d474cb75da730cc73c2ef P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:26:43.079198 4574 raft_consensus.cc:2454] T e733125cd70d474cb75da730cc73c2ef P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:26:43.079602 4375 leader_election.cc:302] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:43.079891 5723 raft_consensus.cc:2781] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:43.080015 5723 raft_consensus.cc:482] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:43.080109 5723 raft_consensus.cc:3037] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:43.081952 5723 raft_consensus.cc:504] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:43.082520 5723 leader_election.cc:288] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:43.082865 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e733125cd70d474cb75da730cc73c2ef" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:26:43.083088 4839 raft_consensus.cc:3037] T e733125cd70d474cb75da730cc73c2ef P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:43.083045 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e733125cd70d474cb75da730cc73c2ef" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:26:43.083288 4574 raft_consensus.cc:3037] T e733125cd70d474cb75da730cc73c2ef P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:43.085637 4839 raft_consensus.cc:2454] T e733125cd70d474cb75da730cc73c2ef P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:26:43.085954 4574 raft_consensus.cc:2454] T e733125cd70d474cb75da730cc73c2ef P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:26:43.085988 4375 leader_election.cc:302] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:43.086670 5723 raft_consensus.cc:2781] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:43.087126 5723 raft_consensus.cc:686] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:26:43.087486 5723 consensus_queue.cc:237] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:43.090452 4294 catalog_manager.cc:5484] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:26:43.100378 6290 raft_consensus.cc:482] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:43.100524 6290 raft_consensus.cc:504] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:43.101296 6290 leader_election.cc:288] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:43.101733 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c49766c76da48b1b35c8837e5816a71" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:43.101787 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c49766c76da48b1b35c8837e5816a71" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:43.102031 4442 raft_consensus.cc:2454] T 1c49766c76da48b1b35c8837e5816a71 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:43.102180 4574 raft_consensus.cc:2454] T 1c49766c76da48b1b35c8837e5816a71 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:43.102468 4770 leader_election.cc:302] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:43.102741 6290 raft_consensus.cc:2781] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:43.102849 6290 raft_consensus.cc:482] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:43.102934 6290 raft_consensus.cc:3037] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:43.104857 6290 raft_consensus.cc:504] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:43.105363 6290 leader_election.cc:288] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:43.105859 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c49766c76da48b1b35c8837e5816a71" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:26:43.105903 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c49766c76da48b1b35c8837e5816a71" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:43.106102 4574 raft_consensus.cc:3037] T 1c49766c76da48b1b35c8837e5816a71 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:43.106086 4442 raft_consensus.cc:3037] T 1c49766c76da48b1b35c8837e5816a71 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:43.108793 4442 raft_consensus.cc:2454] T 1c49766c76da48b1b35c8837e5816a71 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:43.109184 4574 raft_consensus.cc:2454] T 1c49766c76da48b1b35c8837e5816a71 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:43.109179 4770 leader_election.cc:302] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:43.109439 6290 raft_consensus.cc:2781] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:43.109589 6290 raft_consensus.cc:686] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:26:43.109840 6290 consensus_queue.cc:237] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:43.112857 4294 catalog_manager.cc:5484] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:26:43.457684 6058 consensus_queue.cc:1035] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:43.466751 6058 consensus_queue.cc:1035] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:43.481858 6314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:43.482194 6314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:43.490407 6314 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240502 21:26:43.584307 6290 consensus_queue.cc:1035] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:43.598809 5723 consensus_queue.cc:1035] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:43.616477 6286 consensus_queue.cc:1035] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:43.658008 6354 consensus_queue.cc:1035] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 313.888 ms
time per row: 0.156944 ms
I20240502 21:26:43.955014 4291 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:43.964008 4423 tablet_service.cc:1467] Processing CreateTablet for tablet 128dc7838caa403282ed2868444be271 (DEFAULT_TABLE table=rebalance_test_table_11 [id=6578c35615d249838e00b267cf4291d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:43.964164 4684 tablet_service.cc:1467] Processing CreateTablet for tablet 128dc7838caa403282ed2868444be271 (DEFAULT_TABLE table=rebalance_test_table_11 [id=6578c35615d249838e00b267cf4291d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:43.964533 4423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 128dc7838caa403282ed2868444be271. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:43.964655 4684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 128dc7838caa403282ed2868444be271. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:43.965932 4817 tablet_service.cc:1467] Processing CreateTablet for tablet 546a832255ad42bebdf8eaa455ba1c3b (DEFAULT_TABLE table=rebalance_test_table_11 [id=6578c35615d249838e00b267cf4291d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:43.966398 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 546a832255ad42bebdf8eaa455ba1c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:43.968091 4818 tablet_service.cc:1467] Processing CreateTablet for tablet 292bfad92b14456da85bdc874fb7e06e (DEFAULT_TABLE table=rebalance_test_table_11 [id=6578c35615d249838e00b267cf4291d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:43.968494 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 292bfad92b14456da85bdc874fb7e06e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:43.968784 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 546a832255ad42bebdf8eaa455ba1c3b (DEFAULT_TABLE table=rebalance_test_table_11 [id=6578c35615d249838e00b267cf4291d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:43.969135 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 546a832255ad42bebdf8eaa455ba1c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:43.969208 4553 tablet_service.cc:1467] Processing CreateTablet for tablet 546a832255ad42bebdf8eaa455ba1c3b (DEFAULT_TABLE table=rebalance_test_table_11 [id=6578c35615d249838e00b267cf4291d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:43.969640 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 546a832255ad42bebdf8eaa455ba1c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:43.972000 4421 tablet_service.cc:1467] Processing CreateTablet for tablet 292bfad92b14456da85bdc874fb7e06e (DEFAULT_TABLE table=rebalance_test_table_11 [id=6578c35615d249838e00b267cf4291d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:43.972411 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 292bfad92b14456da85bdc874fb7e06e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:43.975977 4685 tablet_service.cc:1467] Processing CreateTablet for tablet 292bfad92b14456da85bdc874fb7e06e (DEFAULT_TABLE table=rebalance_test_table_11 [id=6578c35615d249838e00b267cf4291d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:43.976166 4554 tablet_service.cc:1467] Processing CreateTablet for tablet 128dc7838caa403282ed2868444be271 (DEFAULT_TABLE table=rebalance_test_table_11 [id=6578c35615d249838e00b267cf4291d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:43.976361 4685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 292bfad92b14456da85bdc874fb7e06e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:43.976666 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 128dc7838caa403282ed2868444be271. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:43.990649 6388 tablet_bootstrap.cc:492] T 546a832255ad42bebdf8eaa455ba1c3b P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:43.994912 6388 tablet_bootstrap.cc:654] T 546a832255ad42bebdf8eaa455ba1c3b P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:43.997859 6389 tablet_bootstrap.cc:492] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:43.999194 6392 tablet_bootstrap.cc:492] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:43.999375 6391 tablet_bootstrap.cc:492] T 292bfad92b14456da85bdc874fb7e06e P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:43.999770 6389 tablet_bootstrap.cc:654] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:44.001493 6391 tablet_bootstrap.cc:654] T 292bfad92b14456da85bdc874fb7e06e P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:44.002563 6392 tablet_bootstrap.cc:654] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:44.007040 6388 tablet_bootstrap.cc:492] T 546a832255ad42bebdf8eaa455ba1c3b P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:44.007285 6388 ts_tablet_manager.cc:1367] T 546a832255ad42bebdf8eaa455ba1c3b P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20240502 21:26:44.007373 6391 tablet_bootstrap.cc:492] T 292bfad92b14456da85bdc874fb7e06e P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:44.007539 6391 ts_tablet_manager.cc:1367] T 292bfad92b14456da85bdc874fb7e06e P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240502 21:26:44.008440 6391 raft_consensus.cc:348] T 292bfad92b14456da85bdc874fb7e06e P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:44.008754 6391 raft_consensus.cc:374] T 292bfad92b14456da85bdc874fb7e06e P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:44.008905 6391 raft_consensus.cc:729] T 292bfad92b14456da85bdc874fb7e06e P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:44.009246 6391 consensus_queue.cc:260] T 292bfad92b14456da85bdc874fb7e06e P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:44.010108 6388 raft_consensus.cc:348] T 546a832255ad42bebdf8eaa455ba1c3b P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:44.010399 6388 raft_consensus.cc:374] T 546a832255ad42bebdf8eaa455ba1c3b P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:44.011456 6391 ts_tablet_manager.cc:1398] T 292bfad92b14456da85bdc874fb7e06e P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:26:44.011859 6391 tablet_bootstrap.cc:492] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:44.012210 6388 raft_consensus.cc:729] T 546a832255ad42bebdf8eaa455ba1c3b P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:44.012887 6389 tablet_bootstrap.cc:492] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:44.012749 6388 consensus_queue.cc:260] T 546a832255ad42bebdf8eaa455ba1c3b P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:44.013144 6389 ts_tablet_manager.cc:1367] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.007s
I20240502 21:26:44.013453 6388 ts_tablet_manager.cc:1398] T 546a832255ad42bebdf8eaa455ba1c3b P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 21:26:44.013754 6388 tablet_bootstrap.cc:492] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:44.013983 6389 raft_consensus.cc:348] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:44.014189 6389 raft_consensus.cc:374] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:44.014282 6389 raft_consensus.cc:729] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:44.015429 6388 tablet_bootstrap.cc:654] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:44.017660 6391 tablet_bootstrap.cc:654] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:44.018985 6389 consensus_queue.cc:260] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:44.019605 6391 tablet_bootstrap.cc:492] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:44.019793 6391 ts_tablet_manager.cc:1367] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
I20240502 21:26:44.020021 6388 tablet_bootstrap.cc:492] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:44.020236 6388 ts_tablet_manager.cc:1367] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20240502 21:26:44.020637 6391 raft_consensus.cc:348] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:44.020869 6391 raft_consensus.cc:374] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:44.020960 6391 raft_consensus.cc:729] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:44.021314 6391 consensus_queue.cc:260] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:44.021378 6388 raft_consensus.cc:348] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:44.021870 6391 ts_tablet_manager.cc:1398] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:44.022650 6388 raft_consensus.cc:374] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:44.022796 6388 raft_consensus.cc:729] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:44.023145 6388 consensus_queue.cc:260] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:44.023887 6388 ts_tablet_manager.cc:1398] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:44.024188 6389 ts_tablet_manager.cc:1398] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240502 21:26:44.022218 6391 tablet_bootstrap.cc:492] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:44.026204 6389 tablet_bootstrap.cc:492] T 128dc7838caa403282ed2868444be271 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:44.027801 6389 tablet_bootstrap.cc:654] T 128dc7838caa403282ed2868444be271 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:44.028393 6391 tablet_bootstrap.cc:654] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:44.029263 6392 tablet_bootstrap.cc:492] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:44.029552 6392 ts_tablet_manager.cc:1367] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.003s
I20240502 21:26:44.030419 6392 raft_consensus.cc:348] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:44.030712 6392 raft_consensus.cc:374] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:44.030834 6392 raft_consensus.cc:729] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:44.031312 6392 consensus_queue.cc:260] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:44.032799 6392 ts_tablet_manager.cc:1398] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:44.033138 6392 tablet_bootstrap.cc:492] T 128dc7838caa403282ed2868444be271 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:44.035156 6392 tablet_bootstrap.cc:654] T 128dc7838caa403282ed2868444be271 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:44.043164 6392 tablet_bootstrap.cc:492] T 128dc7838caa403282ed2868444be271 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:44.043385 6392 ts_tablet_manager.cc:1367] T 128dc7838caa403282ed2868444be271 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20240502 21:26:44.044462 6392 raft_consensus.cc:348] T 128dc7838caa403282ed2868444be271 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:44.045171 6391 tablet_bootstrap.cc:492] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:44.045425 6391 ts_tablet_manager.cc:1367] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.009s
I20240502 21:26:44.045672 6389 tablet_bootstrap.cc:492] T 128dc7838caa403282ed2868444be271 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:44.045876 6389 ts_tablet_manager.cc:1367] T 128dc7838caa403282ed2868444be271 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.005s
I20240502 21:26:44.046332 6391 raft_consensus.cc:348] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:44.046497 6392 raft_consensus.cc:374] T 128dc7838caa403282ed2868444be271 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:44.046602 6392 raft_consensus.cc:729] T 128dc7838caa403282ed2868444be271 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:44.046625 6391 raft_consensus.cc:374] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:44.046789 6391 raft_consensus.cc:729] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:44.046856 6392 consensus_queue.cc:260] T 128dc7838caa403282ed2868444be271 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:44.046880 6389 raft_consensus.cc:348] T 128dc7838caa403282ed2868444be271 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:44.047159 6389 raft_consensus.cc:374] T 128dc7838caa403282ed2868444be271 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:44.047276 6389 raft_consensus.cc:729] T 128dc7838caa403282ed2868444be271 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:44.047135 6391 consensus_queue.cc:260] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:44.047473 6392 ts_tablet_manager.cc:1398] T 128dc7838caa403282ed2868444be271 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 21:26:44.048002 6391 ts_tablet_manager.cc:1398] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:44.048390 6389 consensus_queue.cc:260] T 128dc7838caa403282ed2868444be271 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:44.049053 6389 ts_tablet_manager.cc:1398] T 128dc7838caa403282ed2868444be271 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:26:44.073545 6058 raft_consensus.cc:482] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:44.073724 6058 raft_consensus.cc:504] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:44.074546 6058 leader_election.cc:288] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:44.075094 4573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "546a832255ad42bebdf8eaa455ba1c3b" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:44.075095 4837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "546a832255ad42bebdf8eaa455ba1c3b" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:26:44.075330 4837 raft_consensus.cc:2454] T 546a832255ad42bebdf8eaa455ba1c3b P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:26:44.075337 4573 raft_consensus.cc:2454] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:26:44.075752 4642 leader_election.cc:302] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:44.076036 6058 raft_consensus.cc:2781] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:44.076120 6058 raft_consensus.cc:482] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:44.076234 6058 raft_consensus.cc:3037] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:44.078132 6058 raft_consensus.cc:504] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:44.078672 6058 leader_election.cc:288] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Requested vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:44.079092 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "546a832255ad42bebdf8eaa455ba1c3b" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:26:44.079147 4837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "546a832255ad42bebdf8eaa455ba1c3b" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:26:44.079262 4574 raft_consensus.cc:3037] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:44.079362 4837 raft_consensus.cc:3037] T 546a832255ad42bebdf8eaa455ba1c3b P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:44.081539 4574 raft_consensus.cc:2454] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:26:44.081776 4837 raft_consensus.cc:2454] T 546a832255ad42bebdf8eaa455ba1c3b P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:26:44.081996 4642 leader_election.cc:302] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:44.082262 6058 raft_consensus.cc:2781] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:44.082398 6058 raft_consensus.cc:686] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Becoming Leader. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Running, Role: LEADER
I20240502 21:26:44.082620 6058 consensus_queue.cc:237] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:44.085631 4293 catalog_manager.cc:5484] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 398c3c8f3195492c900439ad5e1931bb (127.29.82.67). New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:26:44.146042 5723 raft_consensus.cc:482] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:44.146219 5723 raft_consensus.cc:504] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:44.146878 5723 leader_election.cc:288] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:44.147192 4704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292bfad92b14456da85bdc874fb7e06e" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:44.147305 4837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292bfad92b14456da85bdc874fb7e06e" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:26:44.147428 4704 raft_consensus.cc:2454] T 292bfad92b14456da85bdc874fb7e06e P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:26:44.147560 4837 raft_consensus.cc:2454] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:26:44.147905 4378 leader_election.cc:302] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:26:44.148216 5723 raft_consensus.cc:2781] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:44.148353 5723 raft_consensus.cc:482] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:44.148456 5723 raft_consensus.cc:3037] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:44.150702 5723 raft_consensus.cc:504] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:44.151343 5723 leader_election.cc:288] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:44.151888 4837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292bfad92b14456da85bdc874fb7e06e" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:26:44.151973 4704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292bfad92b14456da85bdc874fb7e06e" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:44.152060 4837 raft_consensus.cc:3037] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:44.152185 4704 raft_consensus.cc:3037] T 292bfad92b14456da85bdc874fb7e06e P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:44.154028 4837 raft_consensus.cc:2454] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:26:44.154243 4704 raft_consensus.cc:2454] T 292bfad92b14456da85bdc874fb7e06e P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:26:44.154605 4378 leader_election.cc:302] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:26:44.154909 5723 raft_consensus.cc:2781] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:44.155073 5723 raft_consensus.cc:686] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:26:44.155340 5723 consensus_queue.cc:237] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:44.158409 4293 catalog_manager.cc:5484] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:26:44.160953 6058 raft_consensus.cc:482] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:44.161125 6058 raft_consensus.cc:504] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:44.161890 6058 leader_election.cc:288] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:44.162204 4441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128dc7838caa403282ed2868444be271" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:44.162321 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128dc7838caa403282ed2868444be271" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:44.162396 4441 raft_consensus.cc:2454] T 128dc7838caa403282ed2868444be271 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:26:44.162571 4574 raft_consensus.cc:2454] T 128dc7838caa403282ed2868444be271 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:26:44.162818 4639 leader_election.cc:302] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:26:44.163069 6058 raft_consensus.cc:2781] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:44.163188 6058 raft_consensus.cc:482] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:44.163290 6058 raft_consensus.cc:3037] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:44.165130 6058 raft_consensus.cc:504] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:44.165724 6058 leader_election.cc:288] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:44.166071 4441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128dc7838caa403282ed2868444be271" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:44.166174 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128dc7838caa403282ed2868444be271" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:26:44.166280 4441 raft_consensus.cc:3037] T 128dc7838caa403282ed2868444be271 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:44.166383 4574 raft_consensus.cc:3037] T 128dc7838caa403282ed2868444be271 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:44.169109 4574 raft_consensus.cc:2454] T 128dc7838caa403282ed2868444be271 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:26:44.169109 4441 raft_consensus.cc:2454] T 128dc7838caa403282ed2868444be271 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:26:44.169481 4639 leader_election.cc:302] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:26:44.169780 6058 raft_consensus.cc:2781] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:44.169914 6058 raft_consensus.cc:686] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Becoming Leader. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Running, Role: LEADER
I20240502 21:26:44.170197 6058 consensus_queue.cc:237] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:44.173081 4291 catalog_manager.cc:5484] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 398c3c8f3195492c900439ad5e1931bb (127.29.82.67). New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
W20240502 21:26:44.572284 6405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:44.572592 6405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:44.580490 6405 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:26:44.592155 6058 consensus_queue.cc:1035] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:44.606676 5723 consensus_queue.cc:1035] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:44.607308 6420 consensus_queue.cc:1035] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:44.617863 5723 consensus_queue.cc:1035] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_11'
I20240502 21:26:44.664319 6420 consensus_queue.cc:1035] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:44.670109 6308 consensus_queue.cc:1035] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 256.197 ms
time per row: 0.128098 ms
I20240502 21:26:45.002091 4293 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:45.012131 4553 tablet_service.cc:1467] Processing CreateTablet for tablet 61f757c73961428b863a29080a8f2652 (DEFAULT_TABLE table=rebalance_test_table_12 [id=26f500760c604cc3aad27dd5cf163804]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:45.012677 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61f757c73961428b863a29080a8f2652. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:45.012882 4421 tablet_service.cc:1467] Processing CreateTablet for tablet 61f757c73961428b863a29080a8f2652 (DEFAULT_TABLE table=rebalance_test_table_12 [id=26f500760c604cc3aad27dd5cf163804]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:45.013298 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61f757c73961428b863a29080a8f2652. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:45.014153 4554 tablet_service.cc:1467] Processing CreateTablet for tablet a2f37797d14b46c59834e9a888e018be (DEFAULT_TABLE table=rebalance_test_table_12 [id=26f500760c604cc3aad27dd5cf163804]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:45.014590 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2f37797d14b46c59834e9a888e018be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:45.016672 4423 tablet_service.cc:1467] Processing CreateTablet for tablet a2f37797d14b46c59834e9a888e018be (DEFAULT_TABLE table=rebalance_test_table_12 [id=26f500760c604cc3aad27dd5cf163804]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:45.017045 4423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2f37797d14b46c59834e9a888e018be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:45.016983 4817 tablet_service.cc:1467] Processing CreateTablet for tablet a2f37797d14b46c59834e9a888e018be (DEFAULT_TABLE table=rebalance_test_table_12 [id=26f500760c604cc3aad27dd5cf163804]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:45.017390 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2f37797d14b46c59834e9a888e018be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:45.020013 4422 tablet_service.cc:1467] Processing CreateTablet for tablet b43650e973c745bda27b050ff5d4c5d7 (DEFAULT_TABLE table=rebalance_test_table_12 [id=26f500760c604cc3aad27dd5cf163804]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:45.020494 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b43650e973c745bda27b050ff5d4c5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:45.020726 4686 tablet_service.cc:1467] Processing CreateTablet for tablet b43650e973c745bda27b050ff5d4c5d7 (DEFAULT_TABLE table=rebalance_test_table_12 [id=26f500760c604cc3aad27dd5cf163804]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:45.021131 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b43650e973c745bda27b050ff5d4c5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:45.023285 4685 tablet_service.cc:1467] Processing CreateTablet for tablet 61f757c73961428b863a29080a8f2652 (DEFAULT_TABLE table=rebalance_test_table_12 [id=26f500760c604cc3aad27dd5cf163804]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:45.023810 4685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61f757c73961428b863a29080a8f2652. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:45.026566 4818 tablet_service.cc:1467] Processing CreateTablet for tablet b43650e973c745bda27b050ff5d4c5d7 (DEFAULT_TABLE table=rebalance_test_table_12 [id=26f500760c604cc3aad27dd5cf163804]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:45.027143 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b43650e973c745bda27b050ff5d4c5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:45.027854 6471 tablet_bootstrap.cc:492] T 61f757c73961428b863a29080a8f2652 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:45.030650 6471 tablet_bootstrap.cc:654] T 61f757c73961428b863a29080a8f2652 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:45.043669 6471 tablet_bootstrap.cc:492] T 61f757c73961428b863a29080a8f2652 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:45.043941 6471 ts_tablet_manager.cc:1367] T 61f757c73961428b863a29080a8f2652 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.010s
I20240502 21:26:45.046110 6471 raft_consensus.cc:348] T 61f757c73961428b863a29080a8f2652 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:45.047019 6471 raft_consensus.cc:374] T 61f757c73961428b863a29080a8f2652 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:45.047044 6473 tablet_bootstrap.cc:492] T b43650e973c745bda27b050ff5d4c5d7 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:45.047247 6471 raft_consensus.cc:729] T 61f757c73961428b863a29080a8f2652 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:45.047634 6471 consensus_queue.cc:260] T 61f757c73961428b863a29080a8f2652 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:45.049023 6471 ts_tablet_manager.cc:1398] T 61f757c73961428b863a29080a8f2652 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:26:45.049435 6471 tablet_bootstrap.cc:492] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:45.051208 6475 tablet_bootstrap.cc:492] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:45.051265 6471 tablet_bootstrap.cc:654] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:45.052927 6473 tablet_bootstrap.cc:654] T b43650e973c745bda27b050ff5d4c5d7 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:45.053205 6475 tablet_bootstrap.cc:654] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:45.055418 6475 tablet_bootstrap.cc:492] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:45.055559 6475 ts_tablet_manager.cc:1367] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 21:26:45.056319 6475 raft_consensus.cc:348] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:45.056515 6475 raft_consensus.cc:374] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:45.056613 6475 raft_consensus.cc:729] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:45.056831 6475 consensus_queue.cc:260] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:45.057375 6475 ts_tablet_manager.cc:1398] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:26:45.057758 6475 tablet_bootstrap.cc:492] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:45.059187 6474 tablet_bootstrap.cc:492] T b43650e973c745bda27b050ff5d4c5d7 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:45.059338 6475 tablet_bootstrap.cc:654] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:45.061463 6471 tablet_bootstrap.cc:492] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:45.061511 6474 tablet_bootstrap.cc:654] T b43650e973c745bda27b050ff5d4c5d7 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:45.061645 6471 ts_tablet_manager.cc:1367] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20240502 21:26:45.062462 6471 raft_consensus.cc:348] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:45.062711 6471 raft_consensus.cc:374] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:45.062815 6471 raft_consensus.cc:729] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:45.063074 6471 consensus_queue.cc:260] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:45.063509 6473 tablet_bootstrap.cc:492] T b43650e973c745bda27b050ff5d4c5d7 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:45.063704 6471 ts_tablet_manager.cc:1398] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:26:45.063738 6473 ts_tablet_manager.cc:1367] T b43650e973c745bda27b050ff5d4c5d7 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240502 21:26:45.065114 6473 raft_consensus.cc:348] T b43650e973c745bda27b050ff5d4c5d7 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:45.065482 6473 raft_consensus.cc:374] T b43650e973c745bda27b050ff5d4c5d7 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:45.065650 6473 raft_consensus.cc:729] T b43650e973c745bda27b050ff5d4c5d7 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:45.065984 6473 consensus_queue.cc:260] T b43650e973c745bda27b050ff5d4c5d7 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:45.066946 6473 ts_tablet_manager.cc:1398] T b43650e973c745bda27b050ff5d4c5d7 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:45.067224 6474 tablet_bootstrap.cc:492] T b43650e973c745bda27b050ff5d4c5d7 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:45.067344 6473 tablet_bootstrap.cc:492] T a2f37797d14b46c59834e9a888e018be P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:45.067421 6474 ts_tablet_manager.cc:1367] T b43650e973c745bda27b050ff5d4c5d7 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20240502 21:26:45.068323 6475 tablet_bootstrap.cc:492] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:45.068526 6475 ts_tablet_manager.cc:1367] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240502 21:26:45.069072 6473 tablet_bootstrap.cc:654] T a2f37797d14b46c59834e9a888e018be P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:45.069365 6475 raft_consensus.cc:348] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:45.069648 6475 raft_consensus.cc:374] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:45.069777 6475 raft_consensus.cc:729] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:45.070040 6474 raft_consensus.cc:348] T b43650e973c745bda27b050ff5d4c5d7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:45.070080 6475 consensus_queue.cc:260] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:45.070329 6474 raft_consensus.cc:374] T b43650e973c745bda27b050ff5d4c5d7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:45.070505 6474 raft_consensus.cc:729] T b43650e973c745bda27b050ff5d4c5d7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:45.070843 6475 ts_tablet_manager.cc:1398] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:45.070823 6474 consensus_queue.cc:260] T b43650e973c745bda27b050ff5d4c5d7 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:45.071534 6474 ts_tablet_manager.cc:1398] T b43650e973c745bda27b050ff5d4c5d7 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:26:45.071961 6474 tablet_bootstrap.cc:492] T a2f37797d14b46c59834e9a888e018be P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:45.073608 6474 tablet_bootstrap.cc:654] T a2f37797d14b46c59834e9a888e018be P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:45.073951 6473 tablet_bootstrap.cc:492] T a2f37797d14b46c59834e9a888e018be P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:45.074168 6473 ts_tablet_manager.cc:1367] T a2f37797d14b46c59834e9a888e018be P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 21:26:45.074977 6473 raft_consensus.cc:348] T a2f37797d14b46c59834e9a888e018be P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:45.075238 6473 raft_consensus.cc:374] T a2f37797d14b46c59834e9a888e018be P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:45.075353 6473 raft_consensus.cc:729] T a2f37797d14b46c59834e9a888e018be P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:45.075671 6473 consensus_queue.cc:260] T a2f37797d14b46c59834e9a888e018be P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:45.076503 6473 ts_tablet_manager.cc:1398] T a2f37797d14b46c59834e9a888e018be P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:45.076876 6473 tablet_bootstrap.cc:492] T 61f757c73961428b863a29080a8f2652 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:45.078575 6473 tablet_bootstrap.cc:654] T 61f757c73961428b863a29080a8f2652 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:45.081861 6474 tablet_bootstrap.cc:492] T a2f37797d14b46c59834e9a888e018be P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:45.082033 6474 ts_tablet_manager.cc:1367] T a2f37797d14b46c59834e9a888e018be P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20240502 21:26:45.082243 6473 tablet_bootstrap.cc:492] T 61f757c73961428b863a29080a8f2652 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:45.082425 6473 ts_tablet_manager.cc:1367] T 61f757c73961428b863a29080a8f2652 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240502 21:26:45.082798 6474 raft_consensus.cc:348] T a2f37797d14b46c59834e9a888e018be P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:45.083009 6474 raft_consensus.cc:374] T a2f37797d14b46c59834e9a888e018be P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:45.083092 6474 raft_consensus.cc:729] T a2f37797d14b46c59834e9a888e018be P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:45.083081 6473 raft_consensus.cc:348] T 61f757c73961428b863a29080a8f2652 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:45.083323 6473 raft_consensus.cc:374] T 61f757c73961428b863a29080a8f2652 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:45.083415 6473 raft_consensus.cc:729] T 61f757c73961428b863a29080a8f2652 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:45.083340 6474 consensus_queue.cc:260] T a2f37797d14b46c59834e9a888e018be P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:45.083678 6473 consensus_queue.cc:260] T 61f757c73961428b863a29080a8f2652 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:45.083889 6474 ts_tablet_manager.cc:1398] T a2f37797d14b46c59834e9a888e018be P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:45.084261 6473 ts_tablet_manager.cc:1398] T 61f757c73961428b863a29080a8f2652 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:26:45.109750 6462 raft_consensus.cc:482] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:45.109918 6462 raft_consensus.cc:504] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:45.110545 6462 leader_election.cc:288] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:45.111069 4837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b43650e973c745bda27b050ff5d4c5d7" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:26:45.111194 4441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b43650e973c745bda27b050ff5d4c5d7" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:45.111346 4837 raft_consensus.cc:2454] T b43650e973c745bda27b050ff5d4c5d7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:26:45.111442 4441 raft_consensus.cc:2454] T b43650e973c745bda27b050ff5d4c5d7 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:26:45.111794 4639 leader_election.cc:302] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:45.112032 6462 raft_consensus.cc:2781] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:45.112151 6462 raft_consensus.cc:482] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:45.112250 6462 raft_consensus.cc:3037] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:45.114399 6462 raft_consensus.cc:504] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:45.114883 6462 leader_election.cc:288] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Requested vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:45.115335 4837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b43650e973c745bda27b050ff5d4c5d7" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:26:45.115396 4441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b43650e973c745bda27b050ff5d4c5d7" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:45.115589 4441 raft_consensus.cc:3037] T b43650e973c745bda27b050ff5d4c5d7 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:45.115563 4837 raft_consensus.cc:3037] T b43650e973c745bda27b050ff5d4c5d7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:45.118453 4441 raft_consensus.cc:2454] T b43650e973c745bda27b050ff5d4c5d7 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:26:45.118439 4837 raft_consensus.cc:2454] T b43650e973c745bda27b050ff5d4c5d7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:26:45.118860 4639 leader_election.cc:302] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:26:45.119174 6462 raft_consensus.cc:2781] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:45.119359 6462 raft_consensus.cc:686] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Becoming Leader. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Running, Role: LEADER
I20240502 21:26:45.119695 6462 consensus_queue.cc:237] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:45.122978 4295 catalog_manager.cc:5484] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 398c3c8f3195492c900439ad5e1931bb (127.29.82.67). New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:26:45.181128 6462 raft_consensus.cc:482] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:45.181352 6462 raft_consensus.cc:504] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:45.181932 6462 leader_election.cc:288] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:45.182351 4441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f757c73961428b863a29080a8f2652" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:45.182346 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f757c73961428b863a29080a8f2652" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:45.182574 4574 raft_consensus.cc:2454] T 61f757c73961428b863a29080a8f2652 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:26:45.182585 4441 raft_consensus.cc:2454] T 61f757c73961428b863a29080a8f2652 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:26:45.182991 4642 leader_election.cc:302] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:45.183287 6462 raft_consensus.cc:2781] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:45.183368 6462 raft_consensus.cc:482] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:45.183454 6462 raft_consensus.cc:3037] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:45.185696 6462 raft_consensus.cc:504] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:45.186443 6462 leader_election.cc:288] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:45.186812 4441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f757c73961428b863a29080a8f2652" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:45.186965 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f757c73961428b863a29080a8f2652" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:26:45.187057 4441 raft_consensus.cc:3037] T 61f757c73961428b863a29080a8f2652 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:45.187160 4574 raft_consensus.cc:3037] T 61f757c73961428b863a29080a8f2652 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:45.189592 4574 raft_consensus.cc:2454] T 61f757c73961428b863a29080a8f2652 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:26:45.189734 4441 raft_consensus.cc:2454] T 61f757c73961428b863a29080a8f2652 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:26:45.189975 4642 leader_election.cc:302] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:45.190227 6462 raft_consensus.cc:2781] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:45.190395 6462 raft_consensus.cc:686] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Becoming Leader. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Running, Role: LEADER
I20240502 21:26:45.190667 6462 consensus_queue.cc:237] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:45.193831 4295 catalog_manager.cc:5484] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 398c3c8f3195492c900439ad5e1931bb (127.29.82.67). New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:26:45.322553 6368 raft_consensus.cc:482] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:45.322785 6368 raft_consensus.cc:504] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:45.323760 6368 leader_election.cc:288] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:45.324333 4837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f37797d14b46c59834e9a888e018be" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:26:45.324393 4441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f37797d14b46c59834e9a888e018be" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:45.324628 4837 raft_consensus.cc:2454] T a2f37797d14b46c59834e9a888e018be P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:26:45.324658 4441 raft_consensus.cc:2454] T a2f37797d14b46c59834e9a888e018be P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:26:45.325085 4508 leader_election.cc:302] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:45.325362 6368 raft_consensus.cc:2781] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:45.325556 6368 raft_consensus.cc:482] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:45.325667 6368 raft_consensus.cc:3037] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:45.327874 6368 raft_consensus.cc:504] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:45.328433 6368 leader_election.cc:288] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:45.328877 4441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f37797d14b46c59834e9a888e018be" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:45.328884 4837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f37797d14b46c59834e9a888e018be" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:26:45.329030 4441 raft_consensus.cc:3037] T a2f37797d14b46c59834e9a888e018be P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:45.329062 4837 raft_consensus.cc:3037] T a2f37797d14b46c59834e9a888e018be P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:45.331429 4441 raft_consensus.cc:2454] T a2f37797d14b46c59834e9a888e018be P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:26:45.331703 4837 raft_consensus.cc:2454] T a2f37797d14b46c59834e9a888e018be P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:26:45.332119 4508 leader_election.cc:302] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:45.332417 6368 raft_consensus.cc:2781] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:45.332770 6368 raft_consensus.cc:686] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:26:45.333029 6368 consensus_queue.cc:237] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:45.336161 4295 catalog_manager.cc:5484] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:26:45.549587 6462 consensus_queue.cc:1035] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:45.556267 6462 consensus_queue.cc:1035] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:45.715337 6368 consensus_queue.cc:1035] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:45.722357 6486 consensus_queue.cc:1035] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:45.768868 6489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:45.769205 6489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 21:26:45.771474 6462 consensus_queue.cc:1035] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:45.779846 6489 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:26:45.779928 6058 consensus_queue.cc:1035] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_12'
INSERT report
rows total: 2000
time total: 225.663 ms
time per row: 0.112832 ms
I20240502 21:26:46.160728 4293 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:46.170980 4554 tablet_service.cc:1467] Processing CreateTablet for tablet 5c5cd4328db94367b4201a49d521a145 (DEFAULT_TABLE table=rebalance_test_table_13 [id=d14fe34dcad44629af56956cc9e71e5b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:46.171087 4818 tablet_service.cc:1467] Processing CreateTablet for tablet 5c5cd4328db94367b4201a49d521a145 (DEFAULT_TABLE table=rebalance_test_table_13 [id=d14fe34dcad44629af56956cc9e71e5b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:46.171504 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c5cd4328db94367b4201a49d521a145. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.171625 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c5cd4328db94367b4201a49d521a145. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.171989 4817 tablet_service.cc:1467] Processing CreateTablet for tablet 81c2664e2d45448b9e08a6fac1c90a0d (DEFAULT_TABLE table=rebalance_test_table_13 [id=d14fe34dcad44629af56956cc9e71e5b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:46.172386 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81c2664e2d45448b9e08a6fac1c90a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.174628 4553 tablet_service.cc:1467] Processing CreateTablet for tablet 81c2664e2d45448b9e08a6fac1c90a0d (DEFAULT_TABLE table=rebalance_test_table_13 [id=d14fe34dcad44629af56956cc9e71e5b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:46.174742 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 5c5cd4328db94367b4201a49d521a145 (DEFAULT_TABLE table=rebalance_test_table_13 [id=d14fe34dcad44629af56956cc9e71e5b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:46.174971 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81c2664e2d45448b9e08a6fac1c90a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.175056 4555 tablet_service.cc:1467] Processing CreateTablet for tablet 6687a533112445dc86951fc778d0725b (DEFAULT_TABLE table=rebalance_test_table_13 [id=d14fe34dcad44629af56956cc9e71e5b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:46.175158 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c5cd4328db94367b4201a49d521a145. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.175405 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6687a533112445dc86951fc778d0725b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.179980 4685 tablet_service.cc:1467] Processing CreateTablet for tablet 81c2664e2d45448b9e08a6fac1c90a0d (DEFAULT_TABLE table=rebalance_test_table_13 [id=d14fe34dcad44629af56956cc9e71e5b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:46.180387 4685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81c2664e2d45448b9e08a6fac1c90a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.184038 4422 tablet_service.cc:1467] Processing CreateTablet for tablet 6687a533112445dc86951fc778d0725b (DEFAULT_TABLE table=rebalance_test_table_13 [id=d14fe34dcad44629af56956cc9e71e5b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:46.184453 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6687a533112445dc86951fc778d0725b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.187752 4684 tablet_service.cc:1467] Processing CreateTablet for tablet 6687a533112445dc86951fc778d0725b (DEFAULT_TABLE table=rebalance_test_table_13 [id=d14fe34dcad44629af56956cc9e71e5b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:46.188122 4684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6687a533112445dc86951fc778d0725b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.190697 6558 tablet_bootstrap.cc:492] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:46.192466 6558 tablet_bootstrap.cc:654] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:46.192871 6557 tablet_bootstrap.cc:492] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:46.197202 6557 tablet_bootstrap.cc:654] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:46.198998 6559 tablet_bootstrap.cc:492] T 5c5cd4328db94367b4201a49d521a145 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:46.202452 6557 tablet_bootstrap.cc:492] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:46.202596 6557 ts_tablet_manager.cc:1367] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20240502 21:26:46.203413 6559 tablet_bootstrap.cc:654] T 5c5cd4328db94367b4201a49d521a145 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:46.203401 6557 raft_consensus.cc:348] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:46.203662 6557 raft_consensus.cc:374] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.203819 6557 raft_consensus.cc:729] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.204187 6562 tablet_bootstrap.cc:492] T 5c5cd4328db94367b4201a49d521a145 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:46.204509 6557 consensus_queue.cc:260] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:46.205430 6557 ts_tablet_manager.cc:1398] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:46.205888 6557 tablet_bootstrap.cc:492] T 6687a533112445dc86951fc778d0725b P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:46.207901 6557 tablet_bootstrap.cc:654] T 6687a533112445dc86951fc778d0725b P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:46.208938 6562 tablet_bootstrap.cc:654] T 5c5cd4328db94367b4201a49d521a145 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:46.216707 6562 tablet_bootstrap.cc:492] T 5c5cd4328db94367b4201a49d521a145 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:46.217038 6562 ts_tablet_manager.cc:1367] T 5c5cd4328db94367b4201a49d521a145 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240502 21:26:46.218127 6557 tablet_bootstrap.cc:492] T 6687a533112445dc86951fc778d0725b P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:46.218176 6558 tablet_bootstrap.cc:492] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:46.218205 6562 raft_consensus.cc:348] T 5c5cd4328db94367b4201a49d521a145 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:46.218374 6557 ts_tablet_manager.cc:1367] T 6687a533112445dc86951fc778d0725b P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20240502 21:26:46.218402 6558 ts_tablet_manager.cc:1367] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.013s
I20240502 21:26:46.218516 6562 raft_consensus.cc:374] T 5c5cd4328db94367b4201a49d521a145 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.218717 6562 raft_consensus.cc:729] T 5c5cd4328db94367b4201a49d521a145 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.219043 6562 consensus_queue.cc:260] T 5c5cd4328db94367b4201a49d521a145 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:46.219242 6557 raft_consensus.cc:348] T 6687a533112445dc86951fc778d0725b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:46.219249 6558 raft_consensus.cc:348] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:46.219439 6558 raft_consensus.cc:374] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.219492 6557 raft_consensus.cc:374] T 6687a533112445dc86951fc778d0725b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.219543 6558 raft_consensus.cc:729] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.219648 6557 raft_consensus.cc:729] T 6687a533112445dc86951fc778d0725b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.219825 6558 consensus_queue.cc:260] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:46.219942 6557 consensus_queue.cc:260] T 6687a533112445dc86951fc778d0725b P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:46.220479 6558 ts_tablet_manager.cc:1398] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:46.221046 6557 ts_tablet_manager.cc:1398] T 6687a533112445dc86951fc778d0725b P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:26:46.222023 6562 ts_tablet_manager.cc:1398] T 5c5cd4328db94367b4201a49d521a145 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:26:46.222357 6562 tablet_bootstrap.cc:492] T 6687a533112445dc86951fc778d0725b P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:46.222375 6557 tablet_bootstrap.cc:492] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:46.224231 6557 tablet_bootstrap.cc:654] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:46.224341 6559 tablet_bootstrap.cc:492] T 5c5cd4328db94367b4201a49d521a145 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:46.224517 6559 ts_tablet_manager.cc:1367] T 5c5cd4328db94367b4201a49d521a145 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.004s
I20240502 21:26:46.225441 6559 raft_consensus.cc:348] T 5c5cd4328db94367b4201a49d521a145 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:46.225667 6559 raft_consensus.cc:374] T 5c5cd4328db94367b4201a49d521a145 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.225785 6559 raft_consensus.cc:729] T 5c5cd4328db94367b4201a49d521a145 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.226027 6559 consensus_queue.cc:260] T 5c5cd4328db94367b4201a49d521a145 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:46.226918 6562 tablet_bootstrap.cc:654] T 6687a533112445dc86951fc778d0725b P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:46.231879 6559 ts_tablet_manager.cc:1398] T 5c5cd4328db94367b4201a49d521a145 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 21:26:46.232251 6559 tablet_bootstrap.cc:492] T 81c2664e2d45448b9e08a6fac1c90a0d P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:46.233124 6557 tablet_bootstrap.cc:492] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:46.233404 6557 ts_tablet_manager.cc:1367] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240502 21:26:46.234198 6559 tablet_bootstrap.cc:654] T 81c2664e2d45448b9e08a6fac1c90a0d P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:46.234625 6557 raft_consensus.cc:348] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:46.234943 6557 raft_consensus.cc:374] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.235114 6557 raft_consensus.cc:729] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.235544 6557 consensus_queue.cc:260] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:46.236371 6557 ts_tablet_manager.cc:1398] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:26:46.244622 6562 tablet_bootstrap.cc:492] T 6687a533112445dc86951fc778d0725b P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:46.244693 6559 tablet_bootstrap.cc:492] T 81c2664e2d45448b9e08a6fac1c90a0d P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:46.244809 6562 ts_tablet_manager.cc:1367] T 6687a533112445dc86951fc778d0725b P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.010s
I20240502 21:26:46.244845 6559 ts_tablet_manager.cc:1367] T 81c2664e2d45448b9e08a6fac1c90a0d P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.001s
I20240502 21:26:46.245469 6562 raft_consensus.cc:348] T 6687a533112445dc86951fc778d0725b P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:46.245519 6559 raft_consensus.cc:348] T 81c2664e2d45448b9e08a6fac1c90a0d P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:46.245657 6562 raft_consensus.cc:374] T 6687a533112445dc86951fc778d0725b P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.245718 6559 raft_consensus.cc:374] T 81c2664e2d45448b9e08a6fac1c90a0d P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.245738 6562 raft_consensus.cc:729] T 6687a533112445dc86951fc778d0725b P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.245790 6559 raft_consensus.cc:729] T 81c2664e2d45448b9e08a6fac1c90a0d P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.246004 6562 consensus_queue.cc:260] T 6687a533112445dc86951fc778d0725b P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:46.246030 6559 consensus_queue.cc:260] T 81c2664e2d45448b9e08a6fac1c90a0d P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:46.246552 6562 ts_tablet_manager.cc:1398] T 6687a533112445dc86951fc778d0725b P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:46.247009 6562 tablet_bootstrap.cc:492] T 81c2664e2d45448b9e08a6fac1c90a0d P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:46.248641 6562 tablet_bootstrap.cc:654] T 81c2664e2d45448b9e08a6fac1c90a0d P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:46.248867 6347 raft_consensus.cc:482] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:46.249015 6347 raft_consensus.cc:504] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:46.249748 6347 leader_election.cc:288] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:46.250672 4704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6687a533112445dc86951fc778d0725b" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:46.250932 4704 raft_consensus.cc:2454] T 6687a533112445dc86951fc778d0725b P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:26:46.251714 4378 leader_election.cc:302] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:26:46.252104 4573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6687a533112445dc86951fc778d0725b" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:46.252316 4573 raft_consensus.cc:2454] T 6687a533112445dc86951fc778d0725b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:26:46.252874 6347 raft_consensus.cc:2781] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:46.252980 6347 raft_consensus.cc:482] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:46.253047 6347 raft_consensus.cc:3037] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:46.255815 6347 raft_consensus.cc:504] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:46.256774 4704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6687a533112445dc86951fc778d0725b" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:46.256976 4704 raft_consensus.cc:3037] T 6687a533112445dc86951fc778d0725b P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:46.259840 4704 raft_consensus.cc:2454] T 6687a533112445dc86951fc778d0725b P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:26:46.260576 4378 leader_election.cc:302] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:26:46.260838 6562 tablet_bootstrap.cc:492] T 81c2664e2d45448b9e08a6fac1c90a0d P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:46.261018 6562 ts_tablet_manager.cc:1367] T 81c2664e2d45448b9e08a6fac1c90a0d P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.012s
I20240502 21:26:46.261801 6347 leader_election.cc:288] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:46.261760 6562 raft_consensus.cc:348] T 81c2664e2d45448b9e08a6fac1c90a0d P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:46.261909 6562 raft_consensus.cc:374] T 81c2664e2d45448b9e08a6fac1c90a0d P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.262010 6562 raft_consensus.cc:729] T 81c2664e2d45448b9e08a6fac1c90a0d P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.262228 6562 consensus_queue.cc:260] T 81c2664e2d45448b9e08a6fac1c90a0d P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:46.262367 6347 raft_consensus.cc:2781] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:46.262504 6347 raft_consensus.cc:686] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:26:46.262765 6562 ts_tablet_manager.cc:1398] T 81c2664e2d45448b9e08a6fac1c90a0d P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:26:46.262741 6347 consensus_queue.cc:237] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:46.267391 4295 catalog_manager.cc:5484] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:26:46.268671 6559 ts_tablet_manager.cc:1398] T 81c2664e2d45448b9e08a6fac1c90a0d P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20240502 21:26:46.270884 6497 raft_consensus.cc:482] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:46.271037 6497 raft_consensus.cc:504] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:46.271924 6497 leader_election.cc:288] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:46.272303 4704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81c2664e2d45448b9e08a6fac1c90a0d" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:46.272492 4704 raft_consensus.cc:2454] T 81c2664e2d45448b9e08a6fac1c90a0d P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:26:46.272554 4837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81c2664e2d45448b9e08a6fac1c90a0d" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:26:46.272775 4837 raft_consensus.cc:2454] T 81c2664e2d45448b9e08a6fac1c90a0d P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:26:46.272869 4511 leader_election.cc:302] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:46.273402 6497 raft_consensus.cc:2781] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:46.273526 6497 raft_consensus.cc:482] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:46.273622 6497 raft_consensus.cc:3037] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:46.274065 4573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6687a533112445dc86951fc778d0725b" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:26:46.274283 4573 raft_consensus.cc:3037] T 6687a533112445dc86951fc778d0725b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:46.276021 6497 raft_consensus.cc:504] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:46.276764 6497 leader_election.cc:288] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:46.277132 4704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81c2664e2d45448b9e08a6fac1c90a0d" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:46.277309 4704 raft_consensus.cc:3037] T 81c2664e2d45448b9e08a6fac1c90a0d P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:46.277299 4837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81c2664e2d45448b9e08a6fac1c90a0d" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:26:46.277413 4573 raft_consensus.cc:2454] T 6687a533112445dc86951fc778d0725b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:26:46.277526 4837 raft_consensus.cc:3037] T 81c2664e2d45448b9e08a6fac1c90a0d P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:46.279727 4837 raft_consensus.cc:2454] T 81c2664e2d45448b9e08a6fac1c90a0d P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:26:46.279820 4704 raft_consensus.cc:2454] T 81c2664e2d45448b9e08a6fac1c90a0d P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:26:46.280182 4508 leader_election.cc:302] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:46.280615 6497 raft_consensus.cc:2781] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:46.280786 6497 raft_consensus.cc:686] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:26:46.281076 6497 consensus_queue.cc:237] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:46.284142 4292 catalog_manager.cc:5484] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:26:46.359869 6497 raft_consensus.cc:482] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:46.360042 6497 raft_consensus.cc:504] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:46.360731 6497 leader_election.cc:288] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:46.361063 4837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c5cd4328db94367b4201a49d521a145" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:26:46.361218 4704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c5cd4328db94367b4201a49d521a145" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:46.361335 4837 raft_consensus.cc:2454] T 5c5cd4328db94367b4201a49d521a145 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:26:46.361464 4704 raft_consensus.cc:2454] T 5c5cd4328db94367b4201a49d521a145 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:26:46.361730 4508 leader_election.cc:302] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:46.361980 6497 raft_consensus.cc:2781] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:46.362072 6497 raft_consensus.cc:482] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:46.362146 6497 raft_consensus.cc:3037] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:46.364096 6497 raft_consensus.cc:504] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:46.364773 6497 leader_election.cc:288] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Requested vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:46.365060 4837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c5cd4328db94367b4201a49d521a145" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:26:46.365291 4837 raft_consensus.cc:3037] T 5c5cd4328db94367b4201a49d521a145 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:46.365499 4704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c5cd4328db94367b4201a49d521a145" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:46.365720 4704 raft_consensus.cc:3037] T 5c5cd4328db94367b4201a49d521a145 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:46.367156 4837 raft_consensus.cc:2454] T 5c5cd4328db94367b4201a49d521a145 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:26:46.367527 4508 leader_election.cc:302] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:46.367686 4704 raft_consensus.cc:2454] T 5c5cd4328db94367b4201a49d521a145 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:26:46.367820 6497 raft_consensus.cc:2781] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:46.368083 6497 raft_consensus.cc:686] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:26:46.368333 6497 consensus_queue.cc:237] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:46.371368 4292 catalog_manager.cc:5484] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:26:46.661307 6497 consensus_queue.cc:1035] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:46.666893 6497 consensus_queue.cc:1035] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:46.691901 6571 consensus_queue.cc:1035] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:46.697742 6571 consensus_queue.cc:1035] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:46.780354 6575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:46.780812 6575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:46.792184 6575 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_13'
I20240502 21:26:46.952198 4705 raft_consensus.cc:1261] T 5c5cd4328db94367b4201a49d521a145 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:46.953006 6368 consensus_queue.cc:1035] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:46.959548 4837 raft_consensus.cc:1261] T 5c5cd4328db94367b4201a49d521a145 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:46.972254 6368 consensus_queue.cc:1035] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 277.009 ms
time per row: 0.138504 ms
I20240502 21:26:47.230419 4293 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:47.240180 4422 tablet_service.cc:1467] Processing CreateTablet for tablet 7e5dd46893db452c88b5c250153674da (DEFAULT_TABLE table=rebalance_test_table_14 [id=54c217fb12554f2689ff67970d9771e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:47.240715 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e5dd46893db452c88b5c250153674da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:47.240584 4553 tablet_service.cc:1467] Processing CreateTablet for tablet 7e5dd46893db452c88b5c250153674da (DEFAULT_TABLE table=rebalance_test_table_14 [id=54c217fb12554f2689ff67970d9771e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:47.241039 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e5dd46893db452c88b5c250153674da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:47.241889 4554 tablet_service.cc:1467] Processing CreateTablet for tablet 69816765eeb84db1871d2873d40215cd (DEFAULT_TABLE table=rebalance_test_table_14 [id=54c217fb12554f2689ff67970d9771e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:47.242278 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69816765eeb84db1871d2873d40215cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:47.243376 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 35da5719247c4901a8521f8cb063a7b1 (DEFAULT_TABLE table=rebalance_test_table_14 [id=54c217fb12554f2689ff67970d9771e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:47.243788 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35da5719247c4901a8521f8cb063a7b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:47.248654 4818 tablet_service.cc:1467] Processing CreateTablet for tablet 7e5dd46893db452c88b5c250153674da (DEFAULT_TABLE table=rebalance_test_table_14 [id=54c217fb12554f2689ff67970d9771e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:47.248955 4816 tablet_service.cc:1467] Processing CreateTablet for tablet 69816765eeb84db1871d2873d40215cd (DEFAULT_TABLE table=rebalance_test_table_14 [id=54c217fb12554f2689ff67970d9771e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:47.249125 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e5dd46893db452c88b5c250153674da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:47.249292 4421 tablet_service.cc:1467] Processing CreateTablet for tablet 35da5719247c4901a8521f8cb063a7b1 (DEFAULT_TABLE table=rebalance_test_table_14 [id=54c217fb12554f2689ff67970d9771e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:47.249409 4816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69816765eeb84db1871d2873d40215cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:47.249702 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35da5719247c4901a8521f8cb063a7b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:47.249851 4817 tablet_service.cc:1467] Processing CreateTablet for tablet 35da5719247c4901a8521f8cb063a7b1 (DEFAULT_TABLE table=rebalance_test_table_14 [id=54c217fb12554f2689ff67970d9771e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:47.250205 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35da5719247c4901a8521f8cb063a7b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:47.253162 4423 tablet_service.cc:1467] Processing CreateTablet for tablet 69816765eeb84db1871d2873d40215cd (DEFAULT_TABLE table=rebalance_test_table_14 [id=54c217fb12554f2689ff67970d9771e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:47.253577 4423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69816765eeb84db1871d2873d40215cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:47.259430 6642 tablet_bootstrap.cc:492] T 69816765eeb84db1871d2873d40215cd P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:47.268587 6642 tablet_bootstrap.cc:654] T 69816765eeb84db1871d2873d40215cd P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:47.269032 6643 tablet_bootstrap.cc:492] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:47.272011 6643 tablet_bootstrap.cc:654] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:47.273588 6640 tablet_bootstrap.cc:492] T 7e5dd46893db452c88b5c250153674da P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:47.273919 6641 tablet_bootstrap.cc:492] T 35da5719247c4901a8521f8cb063a7b1 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:47.275537 6642 tablet_bootstrap.cc:492] T 69816765eeb84db1871d2873d40215cd P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:47.275763 6642 ts_tablet_manager.cc:1367] T 69816765eeb84db1871d2873d40215cd P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.001s
I20240502 21:26:47.276621 6642 raft_consensus.cc:348] T 69816765eeb84db1871d2873d40215cd P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:47.276929 6642 raft_consensus.cc:374] T 69816765eeb84db1871d2873d40215cd P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:47.277021 6641 tablet_bootstrap.cc:654] T 35da5719247c4901a8521f8cb063a7b1 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:47.277055 6642 raft_consensus.cc:729] T 69816765eeb84db1871d2873d40215cd P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:47.277302 6640 tablet_bootstrap.cc:654] T 7e5dd46893db452c88b5c250153674da P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:47.277295 6642 consensus_queue.cc:260] T 69816765eeb84db1871d2873d40215cd P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:47.280014 6643 tablet_bootstrap.cc:492] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:47.281427 6642 ts_tablet_manager.cc:1398] T 69816765eeb84db1871d2873d40215cd P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 21:26:47.283601 6643 ts_tablet_manager.cc:1367] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.005s
I20240502 21:26:47.284070 6642 tablet_bootstrap.cc:492] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:47.284632 6643 raft_consensus.cc:348] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:47.284938 6643 raft_consensus.cc:374] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:47.285038 6641 tablet_bootstrap.cc:492] T 35da5719247c4901a8521f8cb063a7b1 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:47.285091 6643 raft_consensus.cc:729] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:47.285244 6641 ts_tablet_manager.cc:1367] T 35da5719247c4901a8521f8cb063a7b1 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20240502 21:26:47.285540 6643 consensus_queue.cc:260] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:47.285795 6642 tablet_bootstrap.cc:654] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:47.286294 6643 ts_tablet_manager.cc:1398] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:47.286185 6641 raft_consensus.cc:348] T 35da5719247c4901a8521f8cb063a7b1 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:47.286406 6641 raft_consensus.cc:374] T 35da5719247c4901a8521f8cb063a7b1 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:47.286696 6643 tablet_bootstrap.cc:492] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:47.287686 6641 raft_consensus.cc:729] T 35da5719247c4901a8521f8cb063a7b1 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:47.288341 6641 consensus_queue.cc:260] T 35da5719247c4901a8521f8cb063a7b1 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:47.289911 6643 tablet_bootstrap.cc:654] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:47.293856 6641 ts_tablet_manager.cc:1398] T 35da5719247c4901a8521f8cb063a7b1 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 21:26:47.297410 6643 tablet_bootstrap.cc:492] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:47.297643 6643 ts_tablet_manager.cc:1367] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20240502 21:26:47.298520 6643 raft_consensus.cc:348] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:47.300056 6643 raft_consensus.cc:374] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:47.300381 6643 raft_consensus.cc:729] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:47.301183 6643 consensus_queue.cc:260] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:47.302044 6642 tablet_bootstrap.cc:492] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:47.302321 6642 ts_tablet_manager.cc:1367] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.016s
I20240502 21:26:47.302765 6643 ts_tablet_manager.cc:1398] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:26:47.303382 6623 raft_consensus.cc:482] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:47.303411 6642 raft_consensus.cc:348] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:47.303543 6623 raft_consensus.cc:504] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:47.303781 6642 raft_consensus.cc:374] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:47.303900 6642 raft_consensus.cc:729] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:47.304355 6623 leader_election.cc:288] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:47.304479 6642 consensus_queue.cc:260] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:47.305037 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69816765eeb84db1871d2873d40215cd" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:26:47.305049 4441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69816765eeb84db1871d2873d40215cd" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:47.305296 6642 ts_tablet_manager.cc:1398] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:26:47.305265 4838 raft_consensus.cc:2454] T 69816765eeb84db1871d2873d40215cd P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:26:47.305644 6642 tablet_bootstrap.cc:492] T 7e5dd46893db452c88b5c250153674da P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:47.305825 4508 leader_election.cc:302] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
W20240502 21:26:47.306521 4508 leader_election.cc:341] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:47.307040 6623 raft_consensus.cc:2781] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:47.307147 6623 raft_consensus.cc:482] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:47.307215 6623 raft_consensus.cc:3037] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:47.307313 6642 tablet_bootstrap.cc:654] T 7e5dd46893db452c88b5c250153674da P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:47.310312 6623 raft_consensus.cc:504] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:47.310966 6623 leader_election.cc:288] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Requested vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:47.312958 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69816765eeb84db1871d2873d40215cd" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:26:47.313156 4838 raft_consensus.cc:3037] T 69816765eeb84db1871d2873d40215cd P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:47.316431 4838 raft_consensus.cc:2454] T 69816765eeb84db1871d2873d40215cd P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:26:47.317881 4508 leader_election.cc:302] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:47.318519 4441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69816765eeb84db1871d2873d40215cd" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
W20240502 21:26:47.319115 4508 leader_election.cc:341] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:47.322486 6623 raft_consensus.cc:2781] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:47.322950 6623 raft_consensus.cc:686] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:26:47.323273 6623 consensus_queue.cc:237] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:47.327443 6640 tablet_bootstrap.cc:492] T 7e5dd46893db452c88b5c250153674da P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:47.328099 6640 ts_tablet_manager.cc:1367] T 7e5dd46893db452c88b5c250153674da P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.055s user 0.009s sys 0.000s
I20240502 21:26:47.329510 6640 raft_consensus.cc:348] T 7e5dd46893db452c88b5c250153674da P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:47.329934 6640 raft_consensus.cc:374] T 7e5dd46893db452c88b5c250153674da P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:47.330229 6640 raft_consensus.cc:729] T 7e5dd46893db452c88b5c250153674da P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:47.330711 6640 consensus_queue.cc:260] T 7e5dd46893db452c88b5c250153674da P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:47.331729 6640 ts_tablet_manager.cc:1398] T 7e5dd46893db452c88b5c250153674da P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:47.332093 6640 tablet_bootstrap.cc:492] T 69816765eeb84db1871d2873d40215cd P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:47.334348 6640 tablet_bootstrap.cc:654] T 69816765eeb84db1871d2873d40215cd P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:47.334178 4293 catalog_manager.cc:5484] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:26:47.339191 6642 tablet_bootstrap.cc:492] T 7e5dd46893db452c88b5c250153674da P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:47.339346 6642 ts_tablet_manager.cc:1367] T 7e5dd46893db452c88b5c250153674da P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.022s
I20240502 21:26:47.339535 6640 tablet_bootstrap.cc:492] T 69816765eeb84db1871d2873d40215cd P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:47.339717 6640 ts_tablet_manager.cc:1367] T 69816765eeb84db1871d2873d40215cd P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 21:26:47.340318 6640 raft_consensus.cc:348] T 69816765eeb84db1871d2873d40215cd P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:47.340370 6642 raft_consensus.cc:348] T 7e5dd46893db452c88b5c250153674da P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:47.340607 6642 raft_consensus.cc:374] T 7e5dd46893db452c88b5c250153674da P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:47.340642 6640 raft_consensus.cc:374] T 69816765eeb84db1871d2873d40215cd P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:47.340705 6642 raft_consensus.cc:729] T 7e5dd46893db452c88b5c250153674da P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:47.340731 6640 raft_consensus.cc:729] T 69816765eeb84db1871d2873d40215cd P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:47.340934 6642 consensus_queue.cc:260] T 7e5dd46893db452c88b5c250153674da P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:47.340994 6640 consensus_queue.cc:260] T 69816765eeb84db1871d2873d40215cd P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:47.341503 6640 ts_tablet_manager.cc:1398] T 69816765eeb84db1871d2873d40215cd P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:47.341804 6640 tablet_bootstrap.cc:492] T 35da5719247c4901a8521f8cb063a7b1 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:47.342644 6642 ts_tablet_manager.cc:1398] T 7e5dd46893db452c88b5c250153674da P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:47.343433 6640 tablet_bootstrap.cc:654] T 35da5719247c4901a8521f8cb063a7b1 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:47.345167 6640 tablet_bootstrap.cc:492] T 35da5719247c4901a8521f8cb063a7b1 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:47.345309 6640 ts_tablet_manager.cc:1367] T 35da5719247c4901a8521f8cb063a7b1 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 21:26:47.345892 6640 raft_consensus.cc:348] T 35da5719247c4901a8521f8cb063a7b1 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:47.346134 6640 raft_consensus.cc:374] T 35da5719247c4901a8521f8cb063a7b1 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:47.346275 6640 raft_consensus.cc:729] T 35da5719247c4901a8521f8cb063a7b1 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:47.346560 6640 consensus_queue.cc:260] T 35da5719247c4901a8521f8cb063a7b1 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:47.347174 6640 ts_tablet_manager.cc:1398] T 35da5719247c4901a8521f8cb063a7b1 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:47.381934 6623 raft_consensus.cc:482] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:47.382143 6623 raft_consensus.cc:504] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:47.382886 6623 leader_election.cc:288] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:47.383265 4441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e5dd46893db452c88b5c250153674da" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:47.383343 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e5dd46893db452c88b5c250153674da" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:26:47.383544 4441 raft_consensus.cc:2454] T 7e5dd46893db452c88b5c250153674da P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:26:47.383592 4838 raft_consensus.cc:2454] T 7e5dd46893db452c88b5c250153674da P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:26:47.384114 4508 leader_election.cc:302] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:47.384442 6623 raft_consensus.cc:2781] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:47.384594 6623 raft_consensus.cc:482] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:47.384716 6623 raft_consensus.cc:3037] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:47.386833 6623 raft_consensus.cc:504] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:47.387395 6623 leader_election.cc:288] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:47.387912 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e5dd46893db452c88b5c250153674da" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:26:47.387902 4441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e5dd46893db452c88b5c250153674da" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:47.388099 4838 raft_consensus.cc:3037] T 7e5dd46893db452c88b5c250153674da P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:47.388108 4441 raft_consensus.cc:3037] T 7e5dd46893db452c88b5c250153674da P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:47.388839 6584 raft_consensus.cc:482] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:47.389027 6584 raft_consensus.cc:504] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:47.390105 6584 leader_election.cc:288] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:47.390558 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35da5719247c4901a8521f8cb063a7b1" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:47.390767 4443 raft_consensus.cc:2454] T 35da5719247c4901a8521f8cb063a7b1 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:47.390893 4838 raft_consensus.cc:2454] T 7e5dd46893db452c88b5c250153674da P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:26:47.390808 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35da5719247c4901a8521f8cb063a7b1" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:47.391083 4705 raft_consensus.cc:2454] T 35da5719247c4901a8521f8cb063a7b1 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:47.391604 4508 leader_election.cc:302] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:47.391754 4441 raft_consensus.cc:2454] T 7e5dd46893db452c88b5c250153674da P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:26:47.391814 4770 leader_election.cc:302] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:47.391985 6623 raft_consensus.cc:2781] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:47.392145 6623 raft_consensus.cc:686] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:26:47.392191 6584 raft_consensus.cc:2781] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:47.392333 6584 raft_consensus.cc:482] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:47.392419 6584 raft_consensus.cc:3037] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:47.392401 6623 consensus_queue.cc:237] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:47.394722 6584 raft_consensus.cc:504] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:47.395453 6584 leader_election.cc:288] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:47.395747 4441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35da5719247c4901a8521f8cb063a7b1" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:47.395851 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35da5719247c4901a8521f8cb063a7b1" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:47.395943 4441 raft_consensus.cc:3037] T 35da5719247c4901a8521f8cb063a7b1 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:47.396176 4705 raft_consensus.cc:3037] T 35da5719247c4901a8521f8cb063a7b1 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:47.398898 4441 raft_consensus.cc:2454] T 35da5719247c4901a8521f8cb063a7b1 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:47.399122 4705 raft_consensus.cc:2454] T 35da5719247c4901a8521f8cb063a7b1 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:47.399335 4770 leader_election.cc:302] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:47.399664 6584 raft_consensus.cc:2781] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:47.399829 6584 raft_consensus.cc:686] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:26:47.399645 4298 catalog_manager.cc:5484] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:26:47.400125 6584 consensus_queue.cc:237] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:47.403355 4293 catalog_manager.cc:5484] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:26:47.534229 6347 raft_consensus.cc:482] T 69816765eeb84db1871d2873d40215cd P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:47.534490 6347 raft_consensus.cc:504] T 69816765eeb84db1871d2873d40215cd P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:47.535388 6347 leader_election.cc:288] T 69816765eeb84db1871d2873d40215cd P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:47.536118 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69816765eeb84db1871d2873d40215cd" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:47.536096 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69816765eeb84db1871d2873d40215cd" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:26:47.536417 4838 raft_consensus.cc:2379] T 69816765eeb84db1871d2873d40215cd P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 71e1375b799b439fa3c83ca763c039c1 in current term 1: Already voted for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in this term.
I20240502 21:26:47.537091 4375 leader_election.cc:302] T 69816765eeb84db1871d2873d40215cd P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1; no voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:26:47.537595 6347 raft_consensus.cc:3037] T 69816765eeb84db1871d2873d40215cd P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:47.539873 6347 raft_consensus.cc:2726] T 69816765eeb84db1871d2873d40215cd P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:26:47.774947 6623 consensus_queue.cc:1035] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:47.781924 6368 consensus_queue.cc:1035] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:47.789904 6368 consensus_queue.cc:1035] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:47.795074 6656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:47.795650 6656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 21:26:47.796816 6623 consensus_queue.cc:1035] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:47.805399 6451 consensus_queue.cc:1035] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:47.810544 6656 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:26:47.811331 6584 consensus_queue.cc:1035] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_14'
INSERT report
rows total: 2000
time total: 265.739 ms
time per row: 0.13287 ms
I20240502 21:26:48.232020 4299 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:48.241847 4686 tablet_service.cc:1467] Processing CreateTablet for tablet e5ab4fa95c5a46e9a3377a3c96b9e740 (DEFAULT_TABLE table=rebalance_test_table_15 [id=943341d7da6a4071b6595f437f13e626]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:48.241997 4422 tablet_service.cc:1467] Processing CreateTablet for tablet e5ab4fa95c5a46e9a3377a3c96b9e740 (DEFAULT_TABLE table=rebalance_test_table_15 [id=943341d7da6a4071b6595f437f13e626]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:48.242358 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ab4fa95c5a46e9a3377a3c96b9e740. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:48.242462 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ab4fa95c5a46e9a3377a3c96b9e740. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:48.242440 4553 tablet_service.cc:1467] Processing CreateTablet for tablet e5ab4fa95c5a46e9a3377a3c96b9e740 (DEFAULT_TABLE table=rebalance_test_table_15 [id=943341d7da6a4071b6595f437f13e626]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:48.242866 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ab4fa95c5a46e9a3377a3c96b9e740. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:48.244146 4817 tablet_service.cc:1467] Processing CreateTablet for tablet 1d0fcf18fd154a72a3bb2f0744febfc7 (DEFAULT_TABLE table=rebalance_test_table_15 [id=943341d7da6a4071b6595f437f13e626]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:48.244578 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d0fcf18fd154a72a3bb2f0744febfc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:48.247331 4421 tablet_service.cc:1467] Processing CreateTablet for tablet a7a0167853fa420586455b82fb893c46 (DEFAULT_TABLE table=rebalance_test_table_15 [id=943341d7da6a4071b6595f437f13e626]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:48.247473 4818 tablet_service.cc:1467] Processing CreateTablet for tablet a7a0167853fa420586455b82fb893c46 (DEFAULT_TABLE table=rebalance_test_table_15 [id=943341d7da6a4071b6595f437f13e626]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:48.247840 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7a0167853fa420586455b82fb893c46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:48.248030 4685 tablet_service.cc:1467] Processing CreateTablet for tablet a7a0167853fa420586455b82fb893c46 (DEFAULT_TABLE table=rebalance_test_table_15 [id=943341d7da6a4071b6595f437f13e626]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:48.248466 4685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7a0167853fa420586455b82fb893c46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:48.248363 4554 tablet_service.cc:1467] Processing CreateTablet for tablet 1d0fcf18fd154a72a3bb2f0744febfc7 (DEFAULT_TABLE table=rebalance_test_table_15 [id=943341d7da6a4071b6595f437f13e626]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:48.248750 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d0fcf18fd154a72a3bb2f0744febfc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:48.251924 4684 tablet_service.cc:1467] Processing CreateTablet for tablet 1d0fcf18fd154a72a3bb2f0744febfc7 (DEFAULT_TABLE table=rebalance_test_table_15 [id=943341d7da6a4071b6595f437f13e626]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:48.252343 4684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d0fcf18fd154a72a3bb2f0744febfc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:48.254969 6720 tablet_bootstrap.cc:492] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:48.256304 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7a0167853fa420586455b82fb893c46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:48.256851 6720 tablet_bootstrap.cc:654] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:48.259136 6722 tablet_bootstrap.cc:492] T a7a0167853fa420586455b82fb893c46 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:48.261106 6722 tablet_bootstrap.cc:654] T a7a0167853fa420586455b82fb893c46 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:48.267256 6722 tablet_bootstrap.cc:492] T a7a0167853fa420586455b82fb893c46 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:48.267398 6722 ts_tablet_manager.cc:1367] T a7a0167853fa420586455b82fb893c46 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240502 21:26:48.268098 6722 raft_consensus.cc:348] T a7a0167853fa420586455b82fb893c46 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:48.268267 6722 raft_consensus.cc:374] T a7a0167853fa420586455b82fb893c46 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:48.268400 6722 raft_consensus.cc:729] T a7a0167853fa420586455b82fb893c46 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:48.268630 6722 consensus_queue.cc:260] T a7a0167853fa420586455b82fb893c46 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:48.270671 6722 ts_tablet_manager.cc:1398] T a7a0167853fa420586455b82fb893c46 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:26:48.270977 6722 tablet_bootstrap.cc:492] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:48.274195 6719 tablet_bootstrap.cc:492] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:48.274406 6722 tablet_bootstrap.cc:654] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:48.274410 6720 tablet_bootstrap.cc:492] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:48.274556 6720 ts_tablet_manager.cc:1367] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.003s
I20240502 21:26:48.275599 6720 raft_consensus.cc:348] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:48.275892 6720 raft_consensus.cc:374] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:48.276042 6720 raft_consensus.cc:729] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:48.276321 6720 consensus_queue.cc:260] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:48.277020 6720 ts_tablet_manager.cc:1398] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:48.277372 6720 tablet_bootstrap.cc:492] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:48.277407 6724 tablet_bootstrap.cc:492] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:48.278226 6719 tablet_bootstrap.cc:654] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:48.279145 6720 tablet_bootstrap.cc:654] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:48.285113 6720 tablet_bootstrap.cc:492] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:48.285334 6720 ts_tablet_manager.cc:1367] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20240502 21:26:48.286528 6720 raft_consensus.cc:348] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:48.286826 6720 raft_consensus.cc:374] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:48.286947 6720 raft_consensus.cc:729] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:48.287261 6720 consensus_queue.cc:260] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:48.288193 6720 ts_tablet_manager.cc:1398] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:26:48.288615 6724 tablet_bootstrap.cc:654] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:48.292801 6719 tablet_bootstrap.cc:492] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:48.293007 6719 ts_tablet_manager.cc:1367] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.000s
I20240502 21:26:48.293846 6719 raft_consensus.cc:348] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:48.294113 6719 raft_consensus.cc:374] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:48.294270 6719 raft_consensus.cc:729] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:48.294611 6719 consensus_queue.cc:260] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:48.295231 6719 ts_tablet_manager.cc:1398] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:48.295670 6719 tablet_bootstrap.cc:492] T a7a0167853fa420586455b82fb893c46 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:48.297957 6722 tablet_bootstrap.cc:492] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:48.298364 6719 tablet_bootstrap.cc:654] T a7a0167853fa420586455b82fb893c46 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:48.298570 6722 ts_tablet_manager.cc:1367] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.005s
I20240502 21:26:48.299504 6722 raft_consensus.cc:348] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:48.299908 6722 raft_consensus.cc:374] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:48.300050 6722 raft_consensus.cc:729] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:48.300406 6722 consensus_queue.cc:260] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:48.304327 6722 ts_tablet_manager.cc:1398] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 21:26:48.308995 6724 tablet_bootstrap.cc:492] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:48.309125 6719 tablet_bootstrap.cc:492] T a7a0167853fa420586455b82fb893c46 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:48.309172 6724 ts_tablet_manager.cc:1367] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20240502 21:26:48.309289 6719 ts_tablet_manager.cc:1367] T a7a0167853fa420586455b82fb893c46 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.010s
I20240502 21:26:48.309923 6724 raft_consensus.cc:348] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:48.310110 6724 raft_consensus.cc:374] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:48.310043 6719 raft_consensus.cc:348] T a7a0167853fa420586455b82fb893c46 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:48.310230 6724 raft_consensus.cc:729] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:48.310252 6719 raft_consensus.cc:374] T a7a0167853fa420586455b82fb893c46 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:48.310312 6719 raft_consensus.cc:729] T a7a0167853fa420586455b82fb893c46 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:48.310518 6724 consensus_queue.cc:260] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:48.310535 6719 consensus_queue.cc:260] T a7a0167853fa420586455b82fb893c46 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:48.311046 6719 ts_tablet_manager.cc:1398] T a7a0167853fa420586455b82fb893c46 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:26:48.311141 6724 ts_tablet_manager.cc:1398] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:26:48.311362 6719 tablet_bootstrap.cc:492] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:48.312453 6724 tablet_bootstrap.cc:492] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:48.312968 6719 tablet_bootstrap.cc:654] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:48.317732 6719 tablet_bootstrap.cc:492] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:48.317862 6719 ts_tablet_manager.cc:1367] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 21:26:48.318596 6719 raft_consensus.cc:348] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:48.318804 6719 raft_consensus.cc:374] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:48.318881 6719 raft_consensus.cc:729] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:48.319079 6719 consensus_queue.cc:260] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:48.320600 6719 ts_tablet_manager.cc:1398] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:48.323154 6724 tablet_bootstrap.cc:654] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:48.325237 6724 tablet_bootstrap.cc:492] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:48.325577 6724 ts_tablet_manager.cc:1367] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.002s
I20240502 21:26:48.326664 6724 raft_consensus.cc:348] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:48.326862 6724 raft_consensus.cc:374] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:48.326947 6724 raft_consensus.cc:729] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:48.327230 6724 consensus_queue.cc:260] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:48.327803 6724 ts_tablet_manager.cc:1398] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:48.363884 6368 raft_consensus.cc:482] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:48.364068 6368 raft_consensus.cc:504] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:48.364789 6368 leader_election.cc:288] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:48.365142 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ab4fa95c5a46e9a3377a3c96b9e740" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:48.365285 4441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ab4fa95c5a46e9a3377a3c96b9e740" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:48.365406 4705 raft_consensus.cc:2454] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:26:48.365566 4441 raft_consensus.cc:2454] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:26:48.365815 4511 leader_election.cc:302] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:48.366077 6368 raft_consensus.cc:2781] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:48.366197 6368 raft_consensus.cc:482] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:48.366312 6368 raft_consensus.cc:3037] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:48.368575 6368 raft_consensus.cc:504] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:48.369153 6368 leader_election.cc:288] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:48.369482 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ab4fa95c5a46e9a3377a3c96b9e740" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:48.369581 4441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ab4fa95c5a46e9a3377a3c96b9e740" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:48.369688 4705 raft_consensus.cc:3037] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:48.369800 4441 raft_consensus.cc:3037] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:48.371600 4705 raft_consensus.cc:2454] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:26:48.371706 4441 raft_consensus.cc:2454] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:26:48.372014 4511 leader_election.cc:302] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:48.372267 6368 raft_consensus.cc:2781] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:48.372402 6368 raft_consensus.cc:686] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:26:48.372663 6368 consensus_queue.cc:237] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:48.375975 4296 catalog_manager.cc:5484] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:26:48.389308 6701 raft_consensus.cc:482] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:48.389485 6701 raft_consensus.cc:504] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:48.390292 6701 leader_election.cc:288] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:48.390812 4573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d0fcf18fd154a72a3bb2f0744febfc7" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:48.390955 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d0fcf18fd154a72a3bb2f0744febfc7" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:48.391177 4573 raft_consensus.cc:2454] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:48.391202 4705 raft_consensus.cc:2454] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:48.391620 4773 leader_election.cc:302] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:48.391896 6701 raft_consensus.cc:2781] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:48.392020 6701 raft_consensus.cc:482] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:48.392115 6701 raft_consensus.cc:3037] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:48.394335 6701 raft_consensus.cc:504] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:48.394882 6701 leader_election.cc:288] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:48.395551 4573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d0fcf18fd154a72a3bb2f0744febfc7" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:26:48.395685 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d0fcf18fd154a72a3bb2f0744febfc7" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:48.395872 4573 raft_consensus.cc:3037] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:48.395984 4705 raft_consensus.cc:3037] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:48.398718 4573 raft_consensus.cc:2454] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:48.399125 4705 raft_consensus.cc:2454] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:48.399271 4773 leader_election.cc:302] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:48.399677 6701 raft_consensus.cc:2781] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:48.399981 6701 raft_consensus.cc:686] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:26:48.400303 6701 consensus_queue.cc:237] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:48.404739 4296 catalog_manager.cc:5484] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:26:48.422863 6728 raft_consensus.cc:482] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:48.423048 6728 raft_consensus.cc:504] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:48.423893 6728 leader_election.cc:288] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:48.424360 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a0167853fa420586455b82fb893c46" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:26:48.424412 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a0167853fa420586455b82fb893c46" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:48.424580 4838 raft_consensus.cc:2454] T a7a0167853fa420586455b82fb893c46 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:26:48.424647 4705 raft_consensus.cc:2454] T a7a0167853fa420586455b82fb893c46 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:26:48.425033 4375 leader_election.cc:302] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:48.425299 6728 raft_consensus.cc:2781] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:48.425400 6728 raft_consensus.cc:482] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:48.425495 6728 raft_consensus.cc:3037] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:48.427721 6728 raft_consensus.cc:504] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:48.428336 6728 leader_election.cc:288] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:48.428815 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a0167853fa420586455b82fb893c46" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:48.428844 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a0167853fa420586455b82fb893c46" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:26:48.429023 4838 raft_consensus.cc:3037] T a7a0167853fa420586455b82fb893c46 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:48.429033 4705 raft_consensus.cc:3037] T a7a0167853fa420586455b82fb893c46 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:48.431013 4838 raft_consensus.cc:2454] T a7a0167853fa420586455b82fb893c46 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:26:48.431058 4705 raft_consensus.cc:2454] T a7a0167853fa420586455b82fb893c46 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:26:48.431414 4375 leader_election.cc:302] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:48.431705 6728 raft_consensus.cc:2781] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:48.431862 6728 raft_consensus.cc:686] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:26:48.432160 6728 consensus_queue.cc:237] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:48.435640 4296 catalog_manager.cc:5484] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:26:48.763875 6623 consensus_queue.cc:1035] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:48.788851 6735 consensus_queue.cc:1035] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:48.835875 6701 consensus_queue.cc:1035] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:48.845865 6737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:48.846220 6737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 21:26:48.848546 6701 consensus_queue.cc:1035] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:48.856305 6737 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_15'
I20240502 21:26:48.966434 6728 consensus_queue.cc:1035] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:48.972185 6347 consensus_queue.cc:1035] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 302.514 ms
time per row: 0.151257 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:26:50.285480 4295 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:50.308399 4421 tablet_service.cc:1467] Processing CreateTablet for tablet a59e4a0405bc4cac81c9e95bd8d14662 (DEFAULT_TABLE table=rebalance_test_table_16 [id=78a4b23f64324443a6125591bda63185]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:50.308774 4554 tablet_service.cc:1467] Processing CreateTablet for tablet a59e4a0405bc4cac81c9e95bd8d14662 (DEFAULT_TABLE table=rebalance_test_table_16 [id=78a4b23f64324443a6125591bda63185]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:50.308950 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a59e4a0405bc4cac81c9e95bd8d14662. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:50.309240 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a59e4a0405bc4cac81c9e95bd8d14662. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:50.309515 4818 tablet_service.cc:1467] Processing CreateTablet for tablet 8b5d021c9ba34930b2efb654e47ab20d (DEFAULT_TABLE table=rebalance_test_table_16 [id=78a4b23f64324443a6125591bda63185]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:50.309990 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b5d021c9ba34930b2efb654e47ab20d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:50.313292 4422 tablet_service.cc:1467] Processing CreateTablet for tablet 8b5d021c9ba34930b2efb654e47ab20d (DEFAULT_TABLE table=rebalance_test_table_16 [id=78a4b23f64324443a6125591bda63185]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:50.313719 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b5d021c9ba34930b2efb654e47ab20d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:50.314620 4553 tablet_service.cc:1467] Processing CreateTablet for tablet 8b5d021c9ba34930b2efb654e47ab20d (DEFAULT_TABLE table=rebalance_test_table_16 [id=78a4b23f64324443a6125591bda63185]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:50.314625 4684 tablet_service.cc:1467] Processing CreateTablet for tablet 445f931ca1364bb193759d9c6e4ebac9 (DEFAULT_TABLE table=rebalance_test_table_16 [id=78a4b23f64324443a6125591bda63185]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:50.315006 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b5d021c9ba34930b2efb654e47ab20d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:50.315042 4684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 445f931ca1364bb193759d9c6e4ebac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:50.316670 4423 tablet_service.cc:1467] Processing CreateTablet for tablet 445f931ca1364bb193759d9c6e4ebac9 (DEFAULT_TABLE table=rebalance_test_table_16 [id=78a4b23f64324443a6125591bda63185]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:50.317070 4423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 445f931ca1364bb193759d9c6e4ebac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:50.320174 4686 tablet_service.cc:1467] Processing CreateTablet for tablet a59e4a0405bc4cac81c9e95bd8d14662 (DEFAULT_TABLE table=rebalance_test_table_16 [id=78a4b23f64324443a6125591bda63185]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:50.320573 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a59e4a0405bc4cac81c9e95bd8d14662. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:50.321410 4817 tablet_service.cc:1467] Processing CreateTablet for tablet 445f931ca1364bb193759d9c6e4ebac9 (DEFAULT_TABLE table=rebalance_test_table_16 [id=78a4b23f64324443a6125591bda63185]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:50.321820 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 445f931ca1364bb193759d9c6e4ebac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:50.330269 6811 tablet_bootstrap.cc:492] T a59e4a0405bc4cac81c9e95bd8d14662 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:50.332753 6809 tablet_bootstrap.cc:492] T 445f931ca1364bb193759d9c6e4ebac9 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:50.334543 6810 tablet_bootstrap.cc:492] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:50.334623 6811 tablet_bootstrap.cc:654] T a59e4a0405bc4cac81c9e95bd8d14662 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:50.335391 6812 tablet_bootstrap.cc:492] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:50.336282 6809 tablet_bootstrap.cc:654] T 445f931ca1364bb193759d9c6e4ebac9 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:50.336959 6810 tablet_bootstrap.cc:654] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:50.340086 6812 tablet_bootstrap.cc:654] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:50.344599 6809 tablet_bootstrap.cc:492] T 445f931ca1364bb193759d9c6e4ebac9 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:50.344875 6809 ts_tablet_manager.cc:1367] T 445f931ca1364bb193759d9c6e4ebac9 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20240502 21:26:50.345209 6810 tablet_bootstrap.cc:492] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:50.345470 6810 ts_tablet_manager.cc:1367] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20240502 21:26:50.345971 6809 raft_consensus.cc:348] T 445f931ca1364bb193759d9c6e4ebac9 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:50.346398 6809 raft_consensus.cc:374] T 445f931ca1364bb193759d9c6e4ebac9 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:50.346568 6809 raft_consensus.cc:729] T 445f931ca1364bb193759d9c6e4ebac9 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:50.346442 6810 raft_consensus.cc:348] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:50.346696 6810 raft_consensus.cc:374] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:50.346850 6810 raft_consensus.cc:729] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:50.347038 6809 consensus_queue.cc:260] T 445f931ca1364bb193759d9c6e4ebac9 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:50.347091 6810 consensus_queue.cc:260] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:50.347864 6810 ts_tablet_manager.cc:1398] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:50.348341 6810 tablet_bootstrap.cc:492] T 445f931ca1364bb193759d9c6e4ebac9 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:50.349668 6809 ts_tablet_manager.cc:1398] T 445f931ca1364bb193759d9c6e4ebac9 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:26:50.350092 6810 tablet_bootstrap.cc:654] T 445f931ca1364bb193759d9c6e4ebac9 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:50.350311 6809 tablet_bootstrap.cc:492] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:50.352891 6809 tablet_bootstrap.cc:654] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:50.357252 6812 tablet_bootstrap.cc:492] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:50.357367 6809 tablet_bootstrap.cc:492] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:50.357532 6809 ts_tablet_manager.cc:1367] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20240502 21:26:50.357530 6812 ts_tablet_manager.cc:1367] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.008s
I20240502 21:26:50.358287 6809 raft_consensus.cc:348] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:50.358533 6809 raft_consensus.cc:374] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:50.358353 6812 raft_consensus.cc:348] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:50.358628 6809 raft_consensus.cc:729] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:50.358762 6812 raft_consensus.cc:374] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:50.358913 6812 raft_consensus.cc:729] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:50.358932 6809 consensus_queue.cc:260] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:50.359362 6812 consensus_queue.cc:260] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:50.359707 6809 ts_tablet_manager.cc:1398] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 21:26:50.360122 6809 tablet_bootstrap.cc:492] T 8b5d021c9ba34930b2efb654e47ab20d P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:50.360273 6812 ts_tablet_manager.cc:1398] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:26:50.360587 6812 tablet_bootstrap.cc:492] T a59e4a0405bc4cac81c9e95bd8d14662 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:50.361730 6809 tablet_bootstrap.cc:654] T 8b5d021c9ba34930b2efb654e47ab20d P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:50.362376 6812 tablet_bootstrap.cc:654] T a59e4a0405bc4cac81c9e95bd8d14662 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:50.367934 6739 raft_consensus.cc:482] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:50.368141 6739 raft_consensus.cc:504] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:50.369082 6739 leader_election.cc:288] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:50.370978 6812 tablet_bootstrap.cc:492] T a59e4a0405bc4cac81c9e95bd8d14662 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:50.371280 6812 ts_tablet_manager.cc:1367] T a59e4a0405bc4cac81c9e95bd8d14662 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.009s
I20240502 21:26:50.372386 6812 raft_consensus.cc:348] T a59e4a0405bc4cac81c9e95bd8d14662 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:50.372776 6812 raft_consensus.cc:374] T a59e4a0405bc4cac81c9e95bd8d14662 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:50.372927 6812 raft_consensus.cc:729] T a59e4a0405bc4cac81c9e95bd8d14662 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:50.373340 6812 consensus_queue.cc:260] T a59e4a0405bc4cac81c9e95bd8d14662 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:50.374912 6812 ts_tablet_manager.cc:1398] T a59e4a0405bc4cac81c9e95bd8d14662 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:50.375520 4441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445f931ca1364bb193759d9c6e4ebac9" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:50.375839 4441 raft_consensus.cc:2454] T 445f931ca1364bb193759d9c6e4ebac9 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:26:50.376605 4639 leader_election.cc:302] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:26:50.377012 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445f931ca1364bb193759d9c6e4ebac9" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:26:50.378309 6058 raft_consensus.cc:2781] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:50.378465 6058 raft_consensus.cc:482] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:50.378588 6058 raft_consensus.cc:3037] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:26:50.390911 4639 leader_election.cc:341] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:50.392256 6058 raft_consensus.cc:504] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:50.394308 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445f931ca1364bb193759d9c6e4ebac9" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:50.394613 4442 raft_consensus.cc:3037] T 445f931ca1364bb193759d9c6e4ebac9 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:26:50.395790 4773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.67:34537, user_credentials={real_user=slave}} blocked reactor thread for 49538.9us
I20240502 21:26:50.397948 6811 tablet_bootstrap.cc:492] T a59e4a0405bc4cac81c9e95bd8d14662 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:50.398237 6811 ts_tablet_manager.cc:1367] T a59e4a0405bc4cac81c9e95bd8d14662 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.070s user 0.006s sys 0.006s
I20240502 21:26:50.398490 4442 raft_consensus.cc:2454] T 445f931ca1364bb193759d9c6e4ebac9 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:26:50.399273 6811 raft_consensus.cc:348] T a59e4a0405bc4cac81c9e95bd8d14662 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:50.399665 6811 raft_consensus.cc:374] T a59e4a0405bc4cac81c9e95bd8d14662 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:50.399849 6811 raft_consensus.cc:729] T a59e4a0405bc4cac81c9e95bd8d14662 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:50.400202 6811 consensus_queue.cc:260] T a59e4a0405bc4cac81c9e95bd8d14662 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:50.401060 6811 ts_tablet_manager.cc:1398] T a59e4a0405bc4cac81c9e95bd8d14662 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:26:50.401445 6811 tablet_bootstrap.cc:492] T 8b5d021c9ba34930b2efb654e47ab20d P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:50.403457 6811 tablet_bootstrap.cc:654] T 8b5d021c9ba34930b2efb654e47ab20d P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:50.405982 6810 tablet_bootstrap.cc:492] T 445f931ca1364bb193759d9c6e4ebac9 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:50.406237 6810 ts_tablet_manager.cc:1367] T 445f931ca1364bb193759d9c6e4ebac9 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.058s user 0.026s sys 0.030s
I20240502 21:26:50.406656 6809 tablet_bootstrap.cc:492] T 8b5d021c9ba34930b2efb654e47ab20d P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:50.406812 6809 ts_tablet_manager.cc:1367] T 8b5d021c9ba34930b2efb654e47ab20d P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.047s user 0.027s sys 0.009s
I20240502 21:26:50.407289 6810 raft_consensus.cc:348] T 445f931ca1364bb193759d9c6e4ebac9 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:50.407625 6810 raft_consensus.cc:374] T 445f931ca1364bb193759d9c6e4ebac9 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:50.407639 6809 raft_consensus.cc:348] T 8b5d021c9ba34930b2efb654e47ab20d P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:50.407763 6810 raft_consensus.cc:729] T 445f931ca1364bb193759d9c6e4ebac9 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:50.407858 6809 raft_consensus.cc:374] T 8b5d021c9ba34930b2efb654e47ab20d P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:50.407958 6809 raft_consensus.cc:729] T 8b5d021c9ba34930b2efb654e47ab20d P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:50.408174 6809 consensus_queue.cc:260] T 8b5d021c9ba34930b2efb654e47ab20d P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:50.408175 6810 consensus_queue.cc:260] T 445f931ca1364bb193759d9c6e4ebac9 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:50.408862 6809 ts_tablet_manager.cc:1398] T 8b5d021c9ba34930b2efb654e47ab20d P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:50.408913 6810 ts_tablet_manager.cc:1398] T 445f931ca1364bb193759d9c6e4ebac9 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:26:50.410043 6811 tablet_bootstrap.cc:492] T 8b5d021c9ba34930b2efb654e47ab20d P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:50.410262 6811 ts_tablet_manager.cc:1367] T 8b5d021c9ba34930b2efb654e47ab20d P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240502 21:26:50.411106 6811 raft_consensus.cc:348] T 8b5d021c9ba34930b2efb654e47ab20d P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:50.411402 6811 raft_consensus.cc:374] T 8b5d021c9ba34930b2efb654e47ab20d P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:50.411545 6811 raft_consensus.cc:729] T 8b5d021c9ba34930b2efb654e47ab20d P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:50.411854 6811 consensus_queue.cc:260] T 8b5d021c9ba34930b2efb654e47ab20d P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:50.412283 6058 leader_election.cc:288] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:50.412569 6811 ts_tablet_manager.cc:1398] T 8b5d021c9ba34930b2efb654e47ab20d P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:50.415953 4639 leader_election.cc:302] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:26:50.416340 6058 raft_consensus.cc:2781] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:50.416522 6058 raft_consensus.cc:686] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Becoming Leader. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Running, Role: LEADER
I20240502 21:26:50.416734 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445f931ca1364bb193759d9c6e4ebac9" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:26:50.416823 6058 consensus_queue.cc:237] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:50.416965 4839 raft_consensus.cc:3037] T 445f931ca1364bb193759d9c6e4ebac9 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:50.419699 4839 raft_consensus.cc:2454] T 445f931ca1364bb193759d9c6e4ebac9 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:26:50.420404 4295 catalog_manager.cc:5484] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 398c3c8f3195492c900439ad5e1931bb (127.29.82.67). New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:26:50.547343 6817 raft_consensus.cc:482] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:50.547544 6817 raft_consensus.cc:504] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:50.548382 6817 leader_election.cc:288] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:50.549054 4573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b5d021c9ba34930b2efb654e47ab20d" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:50.549458 4573 raft_consensus.cc:2454] T 8b5d021c9ba34930b2efb654e47ab20d P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:50.548820 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b5d021c9ba34930b2efb654e47ab20d" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:50.549690 4442 raft_consensus.cc:2454] T 8b5d021c9ba34930b2efb654e47ab20d P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:50.549984 4773 leader_election.cc:302] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:50.550294 6817 raft_consensus.cc:2781] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:50.550402 6817 raft_consensus.cc:482] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:50.550493 6817 raft_consensus.cc:3037] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:50.552436 6817 raft_consensus.cc:504] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:50.552968 6817 leader_election.cc:288] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:50.553432 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b5d021c9ba34930b2efb654e47ab20d" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:26:50.553416 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b5d021c9ba34930b2efb654e47ab20d" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:50.553617 4574 raft_consensus.cc:3037] T 8b5d021c9ba34930b2efb654e47ab20d P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:50.553617 4442 raft_consensus.cc:3037] T 8b5d021c9ba34930b2efb654e47ab20d P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:50.556440 4442 raft_consensus.cc:2454] T 8b5d021c9ba34930b2efb654e47ab20d P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:50.556440 4574 raft_consensus.cc:2454] T 8b5d021c9ba34930b2efb654e47ab20d P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:50.556902 4770 leader_election.cc:302] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:50.557175 6817 raft_consensus.cc:2781] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:50.557322 6817 raft_consensus.cc:686] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:26:50.557569 6817 consensus_queue.cc:237] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:50.560882 4296 catalog_manager.cc:5484] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:26:50.663059 6347 raft_consensus.cc:482] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:50.663286 6347 raft_consensus.cc:504] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:50.664044 6347 leader_election.cc:288] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:50.664754 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59e4a0405bc4cac81c9e95bd8d14662" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:50.665045 4574 raft_consensus.cc:2454] T a59e4a0405bc4cac81c9e95bd8d14662 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:26:50.665306 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59e4a0405bc4cac81c9e95bd8d14662" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:50.665545 4706 raft_consensus.cc:2454] T a59e4a0405bc4cac81c9e95bd8d14662 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:26:50.665705 4378 leader_election.cc:302] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:50.666071 6825 raft_consensus.cc:2781] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:50.666183 6825 raft_consensus.cc:482] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:50.666301 6825 raft_consensus.cc:3037] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:50.668921 6825 raft_consensus.cc:504] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:50.669425 6825 leader_election.cc:288] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:50.669833 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59e4a0405bc4cac81c9e95bd8d14662" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:26:50.669915 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59e4a0405bc4cac81c9e95bd8d14662" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:50.670032 4574 raft_consensus.cc:3037] T a59e4a0405bc4cac81c9e95bd8d14662 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:50.670126 4705 raft_consensus.cc:3037] T a59e4a0405bc4cac81c9e95bd8d14662 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:50.672057 4574 raft_consensus.cc:2454] T a59e4a0405bc4cac81c9e95bd8d14662 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:26:50.672271 4705 raft_consensus.cc:2454] T a59e4a0405bc4cac81c9e95bd8d14662 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:26:50.672477 4378 leader_election.cc:302] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:50.672773 6825 raft_consensus.cc:2781] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:50.672991 6825 raft_consensus.cc:686] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:26:50.673283 6825 consensus_queue.cc:237] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:50.677368 4295 catalog_manager.cc:5484] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:26:50.818801 6058 consensus_queue.cc:1035] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:50.825897 6058 consensus_queue.cc:1035] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:51.053802 6701 consensus_queue.cc:1035] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:51.058531 6701 consensus_queue.cc:1035] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:51.128729 6827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:51.129037 6827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 21:26:51.132916 6347 consensus_queue.cc:1035] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:51.137849 6347 consensus_queue.cc:1035] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:51.139741 6827 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_16'
INSERT report
rows total: 2000
time total: 243.331 ms
time per row: 0.121665 ms
I20240502 21:26:51.542894 4296 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:51.551996 4554 tablet_service.cc:1467] Processing CreateTablet for tablet 104c676bff964b8b8566b2200ce238c7 (DEFAULT_TABLE table=rebalance_test_table_17 [id=cc837d2089e24e0fa0a7c398ca3eeca8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:51.552567 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 104c676bff964b8b8566b2200ce238c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.553252 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 4dc27c0c9cee447f84e4bc1a63f64b68 (DEFAULT_TABLE table=rebalance_test_table_17 [id=cc837d2089e24e0fa0a7c398ca3eeca8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:51.553719 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc27c0c9cee447f84e4bc1a63f64b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.555691 4684 tablet_service.cc:1467] Processing CreateTablet for tablet 104c676bff964b8b8566b2200ce238c7 (DEFAULT_TABLE table=rebalance_test_table_17 [id=cc837d2089e24e0fa0a7c398ca3eeca8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:51.555750 4818 tablet_service.cc:1467] Processing CreateTablet for tablet 104c676bff964b8b8566b2200ce238c7 (DEFAULT_TABLE table=rebalance_test_table_17 [id=cc837d2089e24e0fa0a7c398ca3eeca8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:51.556098 4684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 104c676bff964b8b8566b2200ce238c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.556208 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 104c676bff964b8b8566b2200ce238c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.557081 4423 tablet_service.cc:1467] Processing CreateTablet for tablet 1c243b27429043aa8642d9b2ed595ab3 (DEFAULT_TABLE table=rebalance_test_table_17 [id=cc837d2089e24e0fa0a7c398ca3eeca8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:51.557502 4423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c243b27429043aa8642d9b2ed595ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.560604 4555 tablet_service.cc:1467] Processing CreateTablet for tablet 1c243b27429043aa8642d9b2ed595ab3 (DEFAULT_TABLE table=rebalance_test_table_17 [id=cc837d2089e24e0fa0a7c398ca3eeca8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:51.560649 4553 tablet_service.cc:1467] Processing CreateTablet for tablet 4dc27c0c9cee447f84e4bc1a63f64b68 (DEFAULT_TABLE table=rebalance_test_table_17 [id=cc837d2089e24e0fa0a7c398ca3eeca8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:51.561012 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c243b27429043aa8642d9b2ed595ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.556094 4685 tablet_service.cc:1467] Processing CreateTablet for tablet 1c243b27429043aa8642d9b2ed595ab3 (DEFAULT_TABLE table=rebalance_test_table_17 [id=cc837d2089e24e0fa0a7c398ca3eeca8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:51.561631 4685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c243b27429043aa8642d9b2ed595ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.564611 4817 tablet_service.cc:1467] Processing CreateTablet for tablet 4dc27c0c9cee447f84e4bc1a63f64b68 (DEFAULT_TABLE table=rebalance_test_table_17 [id=cc837d2089e24e0fa0a7c398ca3eeca8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:51.565078 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc27c0c9cee447f84e4bc1a63f64b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.568363 6890 tablet_bootstrap.cc:492] T 104c676bff964b8b8566b2200ce238c7 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:51.570317 6891 tablet_bootstrap.cc:492] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:51.570695 6890 tablet_bootstrap.cc:654] T 104c676bff964b8b8566b2200ce238c7 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:51.570752 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc27c0c9cee447f84e4bc1a63f64b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.572238 6891 tablet_bootstrap.cc:654] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:51.572556 6892 tablet_bootstrap.cc:492] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:51.574847 6892 tablet_bootstrap.cc:654] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:51.585065 6892 tablet_bootstrap.cc:492] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:51.585333 6892 ts_tablet_manager.cc:1367] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.009s
I20240502 21:26:51.588976 6892 raft_consensus.cc:348] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:51.589308 6892 raft_consensus.cc:374] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:51.589471 6892 raft_consensus.cc:729] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:51.589844 6892 consensus_queue.cc:260] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:51.590587 6892 ts_tablet_manager.cc:1398] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 21:26:51.602290 6891 tablet_bootstrap.cc:492] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:51.602473 6890 tablet_bootstrap.cc:492] T 104c676bff964b8b8566b2200ce238c7 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:51.602607 6891 ts_tablet_manager.cc:1367] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.011s
I20240502 21:26:51.602638 6890 ts_tablet_manager.cc:1367] T 104c676bff964b8b8566b2200ce238c7 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.018s
I20240502 21:26:51.603474 6890 raft_consensus.cc:348] T 104c676bff964b8b8566b2200ce238c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:51.603468 6891 raft_consensus.cc:348] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:51.603715 6890 raft_consensus.cc:374] T 104c676bff964b8b8566b2200ce238c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:51.603768 6891 raft_consensus.cc:374] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:51.603825 6890 raft_consensus.cc:729] T 104c676bff964b8b8566b2200ce238c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:51.603922 6891 raft_consensus.cc:729] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:51.604095 6890 consensus_queue.cc:260] T 104c676bff964b8b8566b2200ce238c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:51.604741 6890 ts_tablet_manager.cc:1398] T 104c676bff964b8b8566b2200ce238c7 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:51.604205 6891 consensus_queue.cc:260] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:51.605449 6891 ts_tablet_manager.cc:1398] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:51.605818 6890 tablet_bootstrap.cc:492] T 1c243b27429043aa8642d9b2ed595ab3 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:51.608450 6890 tablet_bootstrap.cc:654] T 1c243b27429043aa8642d9b2ed595ab3 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:51.605823 6892 tablet_bootstrap.cc:492] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:51.613756 6892 tablet_bootstrap.cc:654] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:51.613934 6896 tablet_bootstrap.cc:492] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:51.616186 6896 tablet_bootstrap.cc:654] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:51.617460 6892 tablet_bootstrap.cc:492] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:51.617636 6892 ts_tablet_manager.cc:1367] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.002s
I20240502 21:26:51.618469 6892 raft_consensus.cc:348] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:51.618717 6892 raft_consensus.cc:374] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:51.618806 6892 raft_consensus.cc:729] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:51.618821 6896 tablet_bootstrap.cc:492] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:51.618966 6896 ts_tablet_manager.cc:1367] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240502 21:26:51.619036 6890 tablet_bootstrap.cc:492] T 1c243b27429043aa8642d9b2ed595ab3 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:51.619212 6890 ts_tablet_manager.cc:1367] T 1c243b27429043aa8642d9b2ed595ab3 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20240502 21:26:51.619081 6892 consensus_queue.cc:260] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:51.619813 6892 ts_tablet_manager.cc:1398] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:51.619792 6896 raft_consensus.cc:348] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:51.619997 6896 raft_consensus.cc:374] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:51.620091 6896 raft_consensus.cc:729] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:51.620004 6890 raft_consensus.cc:348] T 1c243b27429043aa8642d9b2ed595ab3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:51.620211 6890 raft_consensus.cc:374] T 1c243b27429043aa8642d9b2ed595ab3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:51.620281 6890 raft_consensus.cc:729] T 1c243b27429043aa8642d9b2ed595ab3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:51.620311 6896 consensus_queue.cc:260] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:51.620501 6890 consensus_queue.cc:260] T 1c243b27429043aa8642d9b2ed595ab3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:51.621433 6890 ts_tablet_manager.cc:1398] T 1c243b27429043aa8642d9b2ed595ab3 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:26:51.621762 6890 tablet_bootstrap.cc:492] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:51.623411 6890 tablet_bootstrap.cc:654] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:51.624318 6896 ts_tablet_manager.cc:1398] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240502 21:26:51.624660 6896 tablet_bootstrap.cc:492] T 104c676bff964b8b8566b2200ce238c7 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:51.625385 6890 tablet_bootstrap.cc:492] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:51.625516 6890 ts_tablet_manager.cc:1367] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240502 21:26:51.626224 6890 raft_consensus.cc:348] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:51.626437 6896 tablet_bootstrap.cc:654] T 104c676bff964b8b8566b2200ce238c7 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:51.626441 6890 raft_consensus.cc:374] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:51.626545 6890 raft_consensus.cc:729] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:51.626806 6890 consensus_queue.cc:260] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:51.627426 6890 ts_tablet_manager.cc:1398] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:26:51.628800 6896 tablet_bootstrap.cc:492] T 104c676bff964b8b8566b2200ce238c7 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:51.628944 6896 ts_tablet_manager.cc:1367] T 104c676bff964b8b8566b2200ce238c7 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:26:51.629698 6896 raft_consensus.cc:348] T 104c676bff964b8b8566b2200ce238c7 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:51.629889 6896 raft_consensus.cc:374] T 104c676bff964b8b8566b2200ce238c7 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:51.629968 6896 raft_consensus.cc:729] T 104c676bff964b8b8566b2200ce238c7 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:51.630239 6896 consensus_queue.cc:260] T 104c676bff964b8b8566b2200ce238c7 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:51.630746 6896 ts_tablet_manager.cc:1398] T 104c676bff964b8b8566b2200ce238c7 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:51.631227 6896 tablet_bootstrap.cc:492] T 1c243b27429043aa8642d9b2ed595ab3 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:51.632683 6896 tablet_bootstrap.cc:654] T 1c243b27429043aa8642d9b2ed595ab3 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:51.634862 6896 tablet_bootstrap.cc:492] T 1c243b27429043aa8642d9b2ed595ab3 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:51.635007 6896 ts_tablet_manager.cc:1367] T 1c243b27429043aa8642d9b2ed595ab3 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:26:51.635828 6896 raft_consensus.cc:348] T 1c243b27429043aa8642d9b2ed595ab3 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:51.636063 6896 raft_consensus.cc:374] T 1c243b27429043aa8642d9b2ed595ab3 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:51.636168 6896 raft_consensus.cc:729] T 1c243b27429043aa8642d9b2ed595ab3 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:51.636422 6896 consensus_queue.cc:260] T 1c243b27429043aa8642d9b2ed595ab3 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:51.636943 6896 ts_tablet_manager.cc:1398] T 1c243b27429043aa8642d9b2ed595ab3 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:51.640417 6701 raft_consensus.cc:482] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:51.640596 6701 raft_consensus.cc:504] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:51.641206 6701 leader_election.cc:288] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:51.641507 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "104c676bff964b8b8566b2200ce238c7" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:51.641609 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "104c676bff964b8b8566b2200ce238c7" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:51.641716 4574 raft_consensus.cc:2454] T 104c676bff964b8b8566b2200ce238c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:51.641785 4705 raft_consensus.cc:2454] T 104c676bff964b8b8566b2200ce238c7 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:51.642174 4773 leader_election.cc:302] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:51.642446 6701 raft_consensus.cc:2781] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:51.642527 6701 raft_consensus.cc:482] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:51.642645 6701 raft_consensus.cc:3037] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:51.644784 6701 raft_consensus.cc:504] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:51.645326 6701 leader_election.cc:288] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:51.645628 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "104c676bff964b8b8566b2200ce238c7" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:26:51.645795 4574 raft_consensus.cc:3037] T 104c676bff964b8b8566b2200ce238c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:51.645720 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "104c676bff964b8b8566b2200ce238c7" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:51.645926 4705 raft_consensus.cc:3037] T 104c676bff964b8b8566b2200ce238c7 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:51.647996 4574 raft_consensus.cc:2454] T 104c676bff964b8b8566b2200ce238c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:51.648243 4705 raft_consensus.cc:2454] T 104c676bff964b8b8566b2200ce238c7 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:51.648368 4773 leader_election.cc:302] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:51.648658 6701 raft_consensus.cc:2781] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:51.648836 6701 raft_consensus.cc:686] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:26:51.649068 6701 consensus_queue.cc:237] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:51.652128 4296 catalog_manager.cc:5484] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:26:51.680024 6850 raft_consensus.cc:482] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:51.680202 6850 raft_consensus.cc:504] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:51.680902 6850 leader_election.cc:288] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:51.681411 4837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc27c0c9cee447f84e4bc1a63f64b68" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:26:51.681428 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc27c0c9cee447f84e4bc1a63f64b68" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:51.681610 4705 raft_consensus.cc:2454] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:26:51.681638 4837 raft_consensus.cc:2454] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:26:51.682039 4508 leader_election.cc:302] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:51.682370 6850 raft_consensus.cc:2781] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:51.682489 6850 raft_consensus.cc:482] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:51.682595 6850 raft_consensus.cc:3037] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:51.684716 6850 raft_consensus.cc:504] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:51.685233 6850 leader_election.cc:288] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:51.685647 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc27c0c9cee447f84e4bc1a63f64b68" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:51.685689 4837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc27c0c9cee447f84e4bc1a63f64b68" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:26:51.685824 4705 raft_consensus.cc:3037] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:51.685874 4837 raft_consensus.cc:3037] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:51.688356 4705 raft_consensus.cc:2454] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:26:51.688557 4837 raft_consensus.cc:2454] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:26:51.688759 4511 leader_election.cc:302] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:51.689023 6850 raft_consensus.cc:2781] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:51.689178 6850 raft_consensus.cc:686] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:26:51.689453 6850 consensus_queue.cc:237] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:51.692718 4296 catalog_manager.cc:5484] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:26:51.904413 6347 raft_consensus.cc:482] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:51.904601 6347 raft_consensus.cc:504] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:51.905196 6347 leader_election.cc:288] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:51.905627 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c243b27429043aa8642d9b2ed595ab3" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:51.905690 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c243b27429043aa8642d9b2ed595ab3" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:51.905872 4574 raft_consensus.cc:2454] T 1c243b27429043aa8642d9b2ed595ab3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:26:51.905905 4705 raft_consensus.cc:2454] T 1c243b27429043aa8642d9b2ed595ab3 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:26:51.906281 4378 leader_election.cc:302] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:26:51.906555 6347 raft_consensus.cc:2781] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:51.906687 6347 raft_consensus.cc:482] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:51.906791 6347 raft_consensus.cc:3037] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:51.908762 6347 raft_consensus.cc:504] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:51.909369 6347 leader_election.cc:288] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:26:51.909729 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c243b27429043aa8642d9b2ed595ab3" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:26:51.909811 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c243b27429043aa8642d9b2ed595ab3" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:51.909952 4574 raft_consensus.cc:3037] T 1c243b27429043aa8642d9b2ed595ab3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:51.910014 4706 raft_consensus.cc:3037] T 1c243b27429043aa8642d9b2ed595ab3 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:51.912081 4706 raft_consensus.cc:2454] T 1c243b27429043aa8642d9b2ed595ab3 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:26:51.912153 4574 raft_consensus.cc:2454] T 1c243b27429043aa8642d9b2ed595ab3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:26:51.912544 4378 leader_election.cc:302] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:26:51.912832 6347 raft_consensus.cc:2781] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:51.913005 6347 raft_consensus.cc:686] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:26:51.913336 6347 consensus_queue.cc:237] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:51.916426 4296 catalog_manager.cc:5484] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:26:52.075397 6701 consensus_queue.cc:1035] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:52.082301 6701 consensus_queue.cc:1035] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:52.204708 6796 consensus_queue.cc:1035] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:52.210122 6913 consensus_queue.cc:1035] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:52.338203 6906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:52.338644 6906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:52.351200 6906 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_17'
I20240502 21:26:52.561632 6347 consensus_queue.cc:1035] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:52.579944 6825 consensus_queue.cc:1035] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:52.663262 4773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.67:34537, user_credentials={real_user=slave}} blocked reactor thread for 75382us
INSERT report
rows total: 2000
time total: 299.838 ms
time per row: 0.149919 ms
I20240502 21:26:52.816922 4292 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:52.826979 4423 tablet_service.cc:1467] Processing CreateTablet for tablet b16b4fe400714cb99c0b646ca0566329 (DEFAULT_TABLE table=rebalance_test_table_18 [id=41418e80a1734c7e84d8140b5c8fb2c3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:52.827291 4553 tablet_service.cc:1467] Processing CreateTablet for tablet b16b4fe400714cb99c0b646ca0566329 (DEFAULT_TABLE table=rebalance_test_table_18 [id=41418e80a1734c7e84d8140b5c8fb2c3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:52.827528 4423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b16b4fe400714cb99c0b646ca0566329. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:52.827762 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b16b4fe400714cb99c0b646ca0566329. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:52.832042 4817 tablet_service.cc:1467] Processing CreateTablet for tablet b16b4fe400714cb99c0b646ca0566329 (DEFAULT_TABLE table=rebalance_test_table_18 [id=41418e80a1734c7e84d8140b5c8fb2c3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:52.832479 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b16b4fe400714cb99c0b646ca0566329. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:52.832782 4816 tablet_service.cc:1467] Processing CreateTablet for tablet d43765656b5e415886b053e7866dc3b8 (DEFAULT_TABLE table=rebalance_test_table_18 [id=41418e80a1734c7e84d8140b5c8fb2c3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:52.833072 4816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d43765656b5e415886b053e7866dc3b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:52.836028 4555 tablet_service.cc:1467] Processing CreateTablet for tablet d43765656b5e415886b053e7866dc3b8 (DEFAULT_TABLE table=rebalance_test_table_18 [id=41418e80a1734c7e84d8140b5c8fb2c3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:52.836450 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d43765656b5e415886b053e7866dc3b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:52.836419 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 3131b965571144fe9480d59f7607d697 (DEFAULT_TABLE table=rebalance_test_table_18 [id=41418e80a1734c7e84d8140b5c8fb2c3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:52.836977 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3131b965571144fe9480d59f7607d697. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:52.840679 4421 tablet_service.cc:1467] Processing CreateTablet for tablet d43765656b5e415886b053e7866dc3b8 (DEFAULT_TABLE table=rebalance_test_table_18 [id=41418e80a1734c7e84d8140b5c8fb2c3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:52.840761 4422 tablet_service.cc:1467] Processing CreateTablet for tablet 3131b965571144fe9480d59f7607d697 (DEFAULT_TABLE table=rebalance_test_table_18 [id=41418e80a1734c7e84d8140b5c8fb2c3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:52.832388 4818 tablet_service.cc:1467] Processing CreateTablet for tablet 3131b965571144fe9480d59f7607d697 (DEFAULT_TABLE table=rebalance_test_table_18 [id=41418e80a1734c7e84d8140b5c8fb2c3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:52.841140 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d43765656b5e415886b053e7866dc3b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:52.841547 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3131b965571144fe9480d59f7607d697. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:52.848320 6970 tablet_bootstrap.cc:492] T 3131b965571144fe9480d59f7607d697 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:52.848773 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3131b965571144fe9480d59f7607d697. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:52.849027 6969 tablet_bootstrap.cc:492] T d43765656b5e415886b053e7866dc3b8 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:52.850661 6970 tablet_bootstrap.cc:654] T 3131b965571144fe9480d59f7607d697 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:52.851285 6969 tablet_bootstrap.cc:654] T d43765656b5e415886b053e7866dc3b8 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:52.860754 6971 tablet_bootstrap.cc:492] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:52.861579 6970 tablet_bootstrap.cc:492] T 3131b965571144fe9480d59f7607d697 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:52.861806 6970 ts_tablet_manager.cc:1367] T 3131b965571144fe9480d59f7607d697 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20240502 21:26:52.862677 6970 raft_consensus.cc:348] T 3131b965571144fe9480d59f7607d697 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:52.863031 6970 raft_consensus.cc:374] T 3131b965571144fe9480d59f7607d697 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:52.863180 6970 raft_consensus.cc:729] T 3131b965571144fe9480d59f7607d697 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:52.863471 6970 consensus_queue.cc:260] T 3131b965571144fe9480d59f7607d697 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:52.865516 6970 ts_tablet_manager.cc:1398] T 3131b965571144fe9480d59f7607d697 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:26:52.867455 6971 tablet_bootstrap.cc:654] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:52.868077 6968 tablet_bootstrap.cc:492] T b16b4fe400714cb99c0b646ca0566329 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:52.869799 6969 tablet_bootstrap.cc:492] T d43765656b5e415886b053e7866dc3b8 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:52.870007 6969 ts_tablet_manager.cc:1367] T d43765656b5e415886b053e7866dc3b8 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.012s
I20240502 21:26:52.870512 6968 tablet_bootstrap.cc:654] T b16b4fe400714cb99c0b646ca0566329 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:52.871002 6969 raft_consensus.cc:348] T d43765656b5e415886b053e7866dc3b8 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:52.871300 6969 raft_consensus.cc:374] T d43765656b5e415886b053e7866dc3b8 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:52.871433 6969 raft_consensus.cc:729] T d43765656b5e415886b053e7866dc3b8 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:52.871845 6969 consensus_queue.cc:260] T d43765656b5e415886b053e7866dc3b8 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:52.875315 6969 ts_tablet_manager.cc:1398] T d43765656b5e415886b053e7866dc3b8 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:26:52.875497 6971 tablet_bootstrap.cc:492] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:52.875674 6969 tablet_bootstrap.cc:492] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:52.875710 6971 ts_tablet_manager.cc:1367] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.004s
I20240502 21:26:52.875777 6968 tablet_bootstrap.cc:492] T b16b4fe400714cb99c0b646ca0566329 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:52.875953 6968 ts_tablet_manager.cc:1367] T b16b4fe400714cb99c0b646ca0566329 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240502 21:26:52.876600 6971 raft_consensus.cc:348] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:52.876855 6971 raft_consensus.cc:374] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:52.876816 6968 raft_consensus.cc:348] T b16b4fe400714cb99c0b646ca0566329 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:52.876981 6971 raft_consensus.cc:729] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:52.877074 6968 raft_consensus.cc:374] T b16b4fe400714cb99c0b646ca0566329 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:52.877207 6968 raft_consensus.cc:729] T b16b4fe400714cb99c0b646ca0566329 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:52.877262 6971 consensus_queue.cc:260] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:52.877581 6969 tablet_bootstrap.cc:654] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:52.877516 6968 consensus_queue.cc:260] T b16b4fe400714cb99c0b646ca0566329 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:52.877966 6971 ts_tablet_manager.cc:1398] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:52.878163 6968 ts_tablet_manager.cc:1398] T b16b4fe400714cb99c0b646ca0566329 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:52.878275 6971 tablet_bootstrap.cc:492] T b16b4fe400714cb99c0b646ca0566329 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:52.878476 6968 tablet_bootstrap.cc:492] T d43765656b5e415886b053e7866dc3b8 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:52.879878 6971 tablet_bootstrap.cc:654] T b16b4fe400714cb99c0b646ca0566329 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:52.880673 6968 tablet_bootstrap.cc:654] T d43765656b5e415886b053e7866dc3b8 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:52.888579 6971 tablet_bootstrap.cc:492] T b16b4fe400714cb99c0b646ca0566329 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:52.888865 6971 ts_tablet_manager.cc:1367] T b16b4fe400714cb99c0b646ca0566329 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240502 21:26:52.890209 6971 raft_consensus.cc:348] T b16b4fe400714cb99c0b646ca0566329 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:52.890503 6971 raft_consensus.cc:374] T b16b4fe400714cb99c0b646ca0566329 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:52.890640 6971 raft_consensus.cc:729] T b16b4fe400714cb99c0b646ca0566329 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:52.891270 6971 consensus_queue.cc:260] T b16b4fe400714cb99c0b646ca0566329 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:52.892052 6971 ts_tablet_manager.cc:1398] T b16b4fe400714cb99c0b646ca0566329 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:26:52.899924 6971 tablet_bootstrap.cc:492] T 3131b965571144fe9480d59f7607d697 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:52.903478 6971 tablet_bootstrap.cc:654] T 3131b965571144fe9480d59f7607d697 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:52.905867 6971 tablet_bootstrap.cc:492] T 3131b965571144fe9480d59f7607d697 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:52.906024 6971 ts_tablet_manager.cc:1367] T 3131b965571144fe9480d59f7607d697 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240502 21:26:52.906076 6968 tablet_bootstrap.cc:492] T d43765656b5e415886b053e7866dc3b8 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:52.906291 6968 ts_tablet_manager.cc:1367] T d43765656b5e415886b053e7866dc3b8 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.022s
I20240502 21:26:52.906342 6969 tablet_bootstrap.cc:492] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:52.906579 6969 ts_tablet_manager.cc:1367] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.017s
I20240502 21:26:52.906860 6971 raft_consensus.cc:348] T 3131b965571144fe9480d59f7607d697 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:52.907064 6971 raft_consensus.cc:374] T 3131b965571144fe9480d59f7607d697 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:52.906971 6968 raft_consensus.cc:348] T d43765656b5e415886b053e7866dc3b8 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:52.907181 6971 raft_consensus.cc:729] T 3131b965571144fe9480d59f7607d697 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:52.907194 6968 raft_consensus.cc:374] T d43765656b5e415886b053e7866dc3b8 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:52.907287 6968 raft_consensus.cc:729] T d43765656b5e415886b053e7866dc3b8 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:52.907447 6971 consensus_queue.cc:260] T 3131b965571144fe9480d59f7607d697 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:52.907471 6969 raft_consensus.cc:348] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:52.907526 6968 consensus_queue.cc:260] T d43765656b5e415886b053e7866dc3b8 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:52.907711 6969 raft_consensus.cc:374] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:52.908181 6969 raft_consensus.cc:729] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:52.908242 6968 ts_tablet_manager.cc:1398] T d43765656b5e415886b053e7866dc3b8 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:52.908488 6969 consensus_queue.cc:260] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:52.910172 6969 ts_tablet_manager.cc:1398] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:52.910508 6969 tablet_bootstrap.cc:492] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:52.912585 6969 tablet_bootstrap.cc:654] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:52.913805 6971 ts_tablet_manager.cc:1398] T 3131b965571144fe9480d59f7607d697 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:52.915896 6969 tablet_bootstrap.cc:492] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:52.916097 6969 ts_tablet_manager.cc:1367] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 21:26:52.916792 6969 raft_consensus.cc:348] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:52.916950 6969 raft_consensus.cc:374] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:52.917021 6969 raft_consensus.cc:729] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:52.917394 6969 consensus_queue.cc:260] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:52.917948 6969 ts_tablet_manager.cc:1398] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:53.001574 6347 raft_consensus.cc:482] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:53.001847 6347 raft_consensus.cc:504] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:53.002590 6347 leader_election.cc:288] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:53.003026 4572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43765656b5e415886b053e7866dc3b8" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:53.003257 4572 raft_consensus.cc:2454] T d43765656b5e415886b053e7866dc3b8 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:26:53.003412 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43765656b5e415886b053e7866dc3b8" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:26:53.003684 4838 raft_consensus.cc:2454] T d43765656b5e415886b053e7866dc3b8 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:26:53.003888 4378 leader_election.cc:302] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:53.004415 6960 raft_consensus.cc:2781] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:53.004511 6960 raft_consensus.cc:482] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:53.004616 6960 raft_consensus.cc:3037] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:53.006750 6960 raft_consensus.cc:504] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:53.007329 6960 leader_election.cc:288] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:53.007802 4572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43765656b5e415886b053e7866dc3b8" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:26:53.007903 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43765656b5e415886b053e7866dc3b8" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:26:53.008114 4572 raft_consensus.cc:3037] T d43765656b5e415886b053e7866dc3b8 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:53.008138 4838 raft_consensus.cc:3037] T d43765656b5e415886b053e7866dc3b8 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:53.010686 4572 raft_consensus.cc:2454] T d43765656b5e415886b053e7866dc3b8 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:26:53.011065 4838 raft_consensus.cc:2454] T d43765656b5e415886b053e7866dc3b8 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:26:53.011324 4378 leader_election.cc:302] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:53.011659 6960 raft_consensus.cc:2781] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:53.011968 6960 raft_consensus.cc:686] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:26:53.012331 6960 consensus_queue.cc:237] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:53.016640 4298 catalog_manager.cc:5484] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:26:53.061200 6956 raft_consensus.cc:482] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:53.061410 6956 raft_consensus.cc:504] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:53.062203 6956 leader_election.cc:288] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:53.062695 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3131b965571144fe9480d59f7607d697" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:26:53.062815 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3131b965571144fe9480d59f7607d697" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:53.062989 4705 raft_consensus.cc:2454] T 3131b965571144fe9480d59f7607d697 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:53.063068 4443 raft_consensus.cc:2454] T 3131b965571144fe9480d59f7607d697 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:53.063655 4773 leader_election.cc:302] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:53.064024 6956 raft_consensus.cc:2781] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:53.064144 6956 raft_consensus.cc:482] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:53.064267 6956 raft_consensus.cc:3037] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:53.066695 6956 raft_consensus.cc:504] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:53.067221 6956 leader_election.cc:288] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:53.067692 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3131b965571144fe9480d59f7607d697" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:26:53.067706 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3131b965571144fe9480d59f7607d697" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:53.067895 4443 raft_consensus.cc:3037] T 3131b965571144fe9480d59f7607d697 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:53.067916 4705 raft_consensus.cc:3037] T 3131b965571144fe9480d59f7607d697 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:53.070887 4443 raft_consensus.cc:2454] T 3131b965571144fe9480d59f7607d697 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:53.070880 4705 raft_consensus.cc:2454] T 3131b965571144fe9480d59f7607d697 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:53.071363 4773 leader_election.cc:302] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:53.072584 6956 raft_consensus.cc:2781] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:53.072878 6956 raft_consensus.cc:686] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:26:53.073137 6956 consensus_queue.cc:237] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:26:53.076457 4298 catalog_manager.cc:5484] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:26:53.308795 6701 raft_consensus.cc:482] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:53.309007 6701 raft_consensus.cc:504] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:53.309792 6701 leader_election.cc:288] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:53.310355 4572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16b4fe400714cb99c0b646ca0566329" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:53.310372 4441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16b4fe400714cb99c0b646ca0566329" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:53.310616 4572 raft_consensus.cc:2454] T b16b4fe400714cb99c0b646ca0566329 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:53.310619 4441 raft_consensus.cc:2454] T b16b4fe400714cb99c0b646ca0566329 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:53.311048 4773 leader_election.cc:302] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:53.311359 6701 raft_consensus.cc:2781] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:53.311501 6701 raft_consensus.cc:482] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:53.311631 6701 raft_consensus.cc:3037] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:53.313982 6701 raft_consensus.cc:504] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:53.315106 6701 leader_election.cc:288] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:53.315122 4441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16b4fe400714cb99c0b646ca0566329" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:53.315342 4441 raft_consensus.cc:3037] T b16b4fe400714cb99c0b646ca0566329 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:53.315476 4572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16b4fe400714cb99c0b646ca0566329" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:26:53.315732 4572 raft_consensus.cc:3037] T b16b4fe400714cb99c0b646ca0566329 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:53.318475 4572 raft_consensus.cc:2454] T b16b4fe400714cb99c0b646ca0566329 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:53.318499 4441 raft_consensus.cc:2454] T b16b4fe400714cb99c0b646ca0566329 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:53.318913 4773 leader_election.cc:302] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:53.319397 6701 raft_consensus.cc:2781] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:53.319586 6701 raft_consensus.cc:686] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:26:53.319885 6701 consensus_queue.cc:237] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:53.324474 4298 catalog_manager.cc:5484] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:26:53.468322 6960 consensus_queue.cc:1035] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:53.483899 6825 consensus_queue.cc:1035] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:53.618180 6701 consensus_queue.cc:1035] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:53.625274 6993 consensus_queue.cc:1035] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:53.769117 6985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:53.769516 6985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:53.781605 6985 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_18'
I20240502 21:26:53.905731 6992 consensus_queue.cc:1035] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:53.915211 6701 consensus_queue.cc:1035] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 279.141 ms
time per row: 0.139571 ms
I20240502 21:26:54.217017 4298 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:54.227600 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 38ce97f9f9ac4398a16958931ef7262f (DEFAULT_TABLE table=rebalance_test_table_19 [id=879d91ac491a4b2dbed04af44dc5e8de]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:54.228173 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38ce97f9f9ac4398a16958931ef7262f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:54.228490 4553 tablet_service.cc:1467] Processing CreateTablet for tablet 38ce97f9f9ac4398a16958931ef7262f (DEFAULT_TABLE table=rebalance_test_table_19 [id=879d91ac491a4b2dbed04af44dc5e8de]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:54.228519 4685 tablet_service.cc:1467] Processing CreateTablet for tablet b7c60d990751440392ab6e5feb0d8498 (DEFAULT_TABLE table=rebalance_test_table_19 [id=879d91ac491a4b2dbed04af44dc5e8de]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:54.228832 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38ce97f9f9ac4398a16958931ef7262f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:54.228928 4685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7c60d990751440392ab6e5feb0d8498. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:54.232419 4555 tablet_service.cc:1467] Processing CreateTablet for tablet 3a1fadb11cd6436a94e4a67a3aad17ba (DEFAULT_TABLE table=rebalance_test_table_19 [id=879d91ac491a4b2dbed04af44dc5e8de]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:54.232784 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a1fadb11cd6436a94e4a67a3aad17ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:54.233682 7049 tablet_bootstrap.cc:492] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:54.235816 7049 tablet_bootstrap.cc:654] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:54.235833 4422 tablet_service.cc:1467] Processing CreateTablet for tablet 38ce97f9f9ac4398a16958931ef7262f (DEFAULT_TABLE table=rebalance_test_table_19 [id=879d91ac491a4b2dbed04af44dc5e8de]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:54.236404 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38ce97f9f9ac4398a16958931ef7262f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:54.241274 4684 tablet_service.cc:1467] Processing CreateTablet for tablet 3a1fadb11cd6436a94e4a67a3aad17ba (DEFAULT_TABLE table=rebalance_test_table_19 [id=879d91ac491a4b2dbed04af44dc5e8de]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:54.241624 4684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a1fadb11cd6436a94e4a67a3aad17ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:54.242400 4818 tablet_service.cc:1467] Processing CreateTablet for tablet b7c60d990751440392ab6e5feb0d8498 (DEFAULT_TABLE table=rebalance_test_table_19 [id=879d91ac491a4b2dbed04af44dc5e8de]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:54.242807 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7c60d990751440392ab6e5feb0d8498. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:54.243053 4816 tablet_service.cc:1467] Processing CreateTablet for tablet 3a1fadb11cd6436a94e4a67a3aad17ba (DEFAULT_TABLE table=rebalance_test_table_19 [id=879d91ac491a4b2dbed04af44dc5e8de]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:54.243391 4816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a1fadb11cd6436a94e4a67a3aad17ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:54.247226 4423 tablet_service.cc:1467] Processing CreateTablet for tablet b7c60d990751440392ab6e5feb0d8498 (DEFAULT_TABLE table=rebalance_test_table_19 [id=879d91ac491a4b2dbed04af44dc5e8de]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:54.247633 4423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7c60d990751440392ab6e5feb0d8498. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:54.249727 7049 tablet_bootstrap.cc:492] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:54.249926 7049 ts_tablet_manager.cc:1367] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.004s
I20240502 21:26:54.250712 7049 raft_consensus.cc:348] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:54.251039 7049 raft_consensus.cc:374] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:54.251176 7049 raft_consensus.cc:729] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:54.251511 7049 consensus_queue.cc:260] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:54.253244 7049 ts_tablet_manager.cc:1398] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:54.260208 7050 tablet_bootstrap.cc:492] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:54.260943 7052 tablet_bootstrap.cc:492] T b7c60d990751440392ab6e5feb0d8498 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:54.263366 7050 tablet_bootstrap.cc:654] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:54.264283 6913 raft_consensus.cc:482] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:54.264441 6913 raft_consensus.cc:504] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:54.265156 6913 leader_election.cc:288] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
W20240502 21:26:54.266930 4508 leader_election.cc:341] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Not found: Tablet not found: 38ce97f9f9ac4398a16958931ef7262f
I20240502 21:26:54.267098 4704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ce97f9f9ac4398a16958931ef7262f" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
W20240502 21:26:54.267728 4511 leader_election.cc:341] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:54.267882 4511 leader_election.cc:302] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a736a8ebb164ce0a051c0fccd76c3b8; no voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1
I20240502 21:26:54.268379 6796 raft_consensus.cc:2726] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:26:54.269197 7049 tablet_bootstrap.cc:492] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:54.271354 7049 tablet_bootstrap.cc:654] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:54.284468 7052 tablet_bootstrap.cc:654] T b7c60d990751440392ab6e5feb0d8498 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:54.289007 7053 tablet_bootstrap.cc:492] T 3a1fadb11cd6436a94e4a67a3aad17ba P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:54.291100 7053 tablet_bootstrap.cc:654] T 3a1fadb11cd6436a94e4a67a3aad17ba P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:54.291640 7052 tablet_bootstrap.cc:492] T b7c60d990751440392ab6e5feb0d8498 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:54.291798 7052 ts_tablet_manager.cc:1367] T b7c60d990751440392ab6e5feb0d8498 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.002s
I20240502 21:26:54.292573 7049 tablet_bootstrap.cc:492] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:54.292593 7052 raft_consensus.cc:348] T b7c60d990751440392ab6e5feb0d8498 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:54.292759 7049 ts_tablet_manager.cc:1367] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.006s
I20240502 21:26:54.292810 7052 raft_consensus.cc:374] T b7c60d990751440392ab6e5feb0d8498 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:54.292901 7052 raft_consensus.cc:729] T b7c60d990751440392ab6e5feb0d8498 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:54.293138 7052 consensus_queue.cc:260] T b7c60d990751440392ab6e5feb0d8498 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:54.293473 7049 raft_consensus.cc:348] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:54.293686 7049 raft_consensus.cc:374] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:54.293768 7049 raft_consensus.cc:729] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:54.293998 7049 consensus_queue.cc:260] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:54.294606 7049 ts_tablet_manager.cc:1398] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:54.294781 7052 ts_tablet_manager.cc:1398] T b7c60d990751440392ab6e5feb0d8498 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:54.295045 7052 tablet_bootstrap.cc:492] T 38ce97f9f9ac4398a16958931ef7262f P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:54.296651 7052 tablet_bootstrap.cc:654] T 38ce97f9f9ac4398a16958931ef7262f P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:54.298731 7050 tablet_bootstrap.cc:492] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:54.298910 7050 ts_tablet_manager.cc:1367] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.039s user 0.008s sys 0.021s
I20240502 21:26:54.299741 7050 raft_consensus.cc:348] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:54.300137 7052 tablet_bootstrap.cc:492] T 38ce97f9f9ac4398a16958931ef7262f P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:54.300388 7052 ts_tablet_manager.cc:1367] T 38ce97f9f9ac4398a16958931ef7262f P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:26:54.299978 7050 raft_consensus.cc:374] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:54.300879 7050 raft_consensus.cc:729] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:54.301139 7050 consensus_queue.cc:260] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:54.301712 7050 ts_tablet_manager.cc:1398] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:54.302043 7050 tablet_bootstrap.cc:492] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:54.302345 7052 raft_consensus.cc:348] T 38ce97f9f9ac4398a16958931ef7262f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:54.302611 7052 raft_consensus.cc:374] T 38ce97f9f9ac4398a16958931ef7262f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:54.302731 7052 raft_consensus.cc:729] T 38ce97f9f9ac4398a16958931ef7262f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:54.303037 7052 consensus_queue.cc:260] T 38ce97f9f9ac4398a16958931ef7262f P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:54.303851 7052 ts_tablet_manager.cc:1398] T 38ce97f9f9ac4398a16958931ef7262f P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:26:54.304056 7050 tablet_bootstrap.cc:654] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:54.321074 7050 tablet_bootstrap.cc:492] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:54.321292 7050 ts_tablet_manager.cc:1367] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.013s
I20240502 21:26:54.322249 7053 tablet_bootstrap.cc:492] T 3a1fadb11cd6436a94e4a67a3aad17ba P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:54.322465 7053 ts_tablet_manager.cc:1367] T 3a1fadb11cd6436a94e4a67a3aad17ba P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.007s
I20240502 21:26:54.322628 7050 raft_consensus.cc:348] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:54.322885 7050 raft_consensus.cc:374] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:54.322981 7050 raft_consensus.cc:729] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:54.323391 7053 raft_consensus.cc:348] T 3a1fadb11cd6436a94e4a67a3aad17ba P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:54.323689 7053 raft_consensus.cc:374] T 3a1fadb11cd6436a94e4a67a3aad17ba P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:54.323405 7050 consensus_queue.cc:260] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:54.323879 7053 raft_consensus.cc:729] T 3a1fadb11cd6436a94e4a67a3aad17ba P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:54.324227 7050 ts_tablet_manager.cc:1398] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:54.324136 7053 consensus_queue.cc:260] T 3a1fadb11cd6436a94e4a67a3aad17ba P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:54.324640 7050 tablet_bootstrap.cc:492] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:26:54.324853 7053 ts_tablet_manager.cc:1398] T 3a1fadb11cd6436a94e4a67a3aad17ba P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:54.325199 7053 tablet_bootstrap.cc:492] T b7c60d990751440392ab6e5feb0d8498 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:54.326300 7050 tablet_bootstrap.cc:654] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:54.326930 7053 tablet_bootstrap.cc:654] T b7c60d990751440392ab6e5feb0d8498 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:54.344583 7050 tablet_bootstrap.cc:492] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:26:54.344630 7053 tablet_bootstrap.cc:492] T b7c60d990751440392ab6e5feb0d8498 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:54.344743 7050 ts_tablet_manager.cc:1367] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.008s
I20240502 21:26:54.344781 7053 ts_tablet_manager.cc:1367] T b7c60d990751440392ab6e5feb0d8498 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.005s
I20240502 21:26:54.345335 7050 raft_consensus.cc:348] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:54.345495 7050 raft_consensus.cc:374] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:54.345446 7053 raft_consensus.cc:348] T b7c60d990751440392ab6e5feb0d8498 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:54.345589 7050 raft_consensus.cc:729] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:26:54.345679 7053 raft_consensus.cc:374] T b7c60d990751440392ab6e5feb0d8498 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:54.345793 7053 raft_consensus.cc:729] T b7c60d990751440392ab6e5feb0d8498 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:54.345814 7050 consensus_queue.cc:260] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:54.346029 7053 consensus_queue.cc:260] T b7c60d990751440392ab6e5feb0d8498 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:54.346526 7050 ts_tablet_manager.cc:1398] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:54.346540 7053 ts_tablet_manager.cc:1398] T b7c60d990751440392ab6e5feb0d8498 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:54.397073 6930 raft_consensus.cc:482] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:54.397284 6930 raft_consensus.cc:504] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:54.398020 6930 leader_election.cc:288] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:54.398587 4572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a1fadb11cd6436a94e4a67a3aad17ba" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:54.398630 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a1fadb11cd6436a94e4a67a3aad17ba" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:26:54.398833 4572 raft_consensus.cc:2454] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:26:54.398882 4839 raft_consensus.cc:2454] T 3a1fadb11cd6436a94e4a67a3aad17ba P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:26:54.399377 4639 leader_election.cc:302] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:54.399873 6930 raft_consensus.cc:2781] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:54.399992 6930 raft_consensus.cc:482] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:54.400085 6930 raft_consensus.cc:3037] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:54.402647 6930 raft_consensus.cc:504] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:54.403306 6930 leader_election.cc:288] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Requested vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:54.404135 4572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a1fadb11cd6436a94e4a67a3aad17ba" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:26:54.404358 4572 raft_consensus.cc:3037] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:54.404700 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a1fadb11cd6436a94e4a67a3aad17ba" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:26:54.404937 4839 raft_consensus.cc:3037] T 3a1fadb11cd6436a94e4a67a3aad17ba P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:54.406873 4572 raft_consensus.cc:2454] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:26:54.407156 4839 raft_consensus.cc:2454] T 3a1fadb11cd6436a94e4a67a3aad17ba P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:26:54.407503 4639 leader_election.cc:302] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:54.407842 6930 raft_consensus.cc:2781] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:54.408290 6930 raft_consensus.cc:686] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Becoming Leader. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Running, Role: LEADER
I20240502 21:26:54.408552 6930 consensus_queue.cc:237] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:26:54.412212 4292 catalog_manager.cc:5484] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 398c3c8f3195492c900439ad5e1931bb (127.29.82.67). New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:26:54.449390 6930 raft_consensus.cc:482] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:54.449594 6930 raft_consensus.cc:504] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:54.450188 6930 leader_election.cc:288] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:54.450663 4572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ce97f9f9ac4398a16958931ef7262f" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:54.450675 4441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ce97f9f9ac4398a16958931ef7262f" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:54.450914 4572 raft_consensus.cc:2454] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:26:54.450935 4441 raft_consensus.cc:2454] T 38ce97f9f9ac4398a16958931ef7262f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:26:54.451293 4642 leader_election.cc:302] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:54.451710 6930 raft_consensus.cc:2781] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:54.451797 6930 raft_consensus.cc:482] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:54.451908 6930 raft_consensus.cc:3037] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:54.453752 6930 raft_consensus.cc:504] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:54.454273 6930 leader_election.cc:288] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:26:54.454638 4572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ce97f9f9ac4398a16958931ef7262f" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:26:54.454674 4441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ce97f9f9ac4398a16958931ef7262f" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:54.454866 4572 raft_consensus.cc:3037] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:54.454885 4441 raft_consensus.cc:3037] T 38ce97f9f9ac4398a16958931ef7262f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:54.458029 4441 raft_consensus.cc:2454] T 38ce97f9f9ac4398a16958931ef7262f P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:26:54.458176 4572 raft_consensus.cc:2454] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:26:54.458619 4642 leader_election.cc:302] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:54.458940 6930 raft_consensus.cc:2781] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:54.459198 6930 raft_consensus.cc:686] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Becoming Leader. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Running, Role: LEADER
I20240502 21:26:54.459600 6930 consensus_queue.cc:237] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:26:54.463356 4292 catalog_manager.cc:5484] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 398c3c8f3195492c900439ad5e1931bb (127.29.82.67). New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:26:54.468345 6930 raft_consensus.cc:482] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:54.468492 6930 raft_consensus.cc:504] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:54.469028 6930 leader_election.cc:288] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:54.469373 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c60d990751440392ab6e5feb0d8498" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:26:54.469472 4441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c60d990751440392ab6e5feb0d8498" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:54.469641 4839 raft_consensus.cc:2454] T b7c60d990751440392ab6e5feb0d8498 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:26:54.469708 4441 raft_consensus.cc:2454] T b7c60d990751440392ab6e5feb0d8498 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:26:54.470036 4639 leader_election.cc:302] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:54.470306 6930 raft_consensus.cc:2781] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:54.470479 6930 raft_consensus.cc:482] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:54.470641 6930 raft_consensus.cc:3037] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:54.472571 6930 raft_consensus.cc:504] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:54.473073 6930 leader_election.cc:288] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Requested vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:54.473433 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c60d990751440392ab6e5feb0d8498" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:26:54.473663 4839 raft_consensus.cc:3037] T b7c60d990751440392ab6e5feb0d8498 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:54.473634 4441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c60d990751440392ab6e5feb0d8498" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:54.473837 4441 raft_consensus.cc:3037] T b7c60d990751440392ab6e5feb0d8498 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:54.476485 4839 raft_consensus.cc:2454] T b7c60d990751440392ab6e5feb0d8498 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:26:54.476611 4441 raft_consensus.cc:2454] T b7c60d990751440392ab6e5feb0d8498 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:26:54.476900 4639 leader_election.cc:302] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:54.477186 6930 raft_consensus.cc:2781] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:54.477340 6930 raft_consensus.cc:686] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Becoming Leader. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Running, Role: LEADER
I20240502 21:26:54.477609 6930 consensus_queue.cc:237] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:26:54.480655 4292 catalog_manager.cc:5484] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 398c3c8f3195492c900439ad5e1931bb (127.29.82.67). New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
W20240502 21:26:54.866101 7066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:54.866403 7066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:54.873639 7066 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:26:54.892139 7054 consensus_queue.cc:1035] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:54.896263 7054 consensus_queue.cc:1035] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_19'
I20240502 21:26:54.983501 6930 consensus_queue.cc:1035] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:55.006887 6930 consensus_queue.cc:1035] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:55.051442 4443 raft_consensus.cc:1261] T b7c60d990751440392ab6e5feb0d8498 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:55.052269 6930 consensus_queue.cc:1035] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:55.053586 4838 raft_consensus.cc:1261] T b7c60d990751440392ab6e5feb0d8498 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:55.054252 6930 consensus_queue.cc:1035] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:55.191720 7107 mvcc.cc:205] Tried to move back new op lower bound from 7023350640840249344 to 7023350638502166528. Current Snapshot: MvccSnapshot[applied={T|T < 7023350640840249344}]
INSERT report
rows total: 2000
time total: 282.603 ms
time per row: 0.141301 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:26:56.296260 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.69:33193
--local_ip_for_outbound_sockets=127.29.82.69
--tserver_master_addrs=127.29.82.126:37445
--webserver_port=40947
--webserver_interface=127.29.82.69
--builtin_ntp_servers=127.29.82.84:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 21:26:56.651055 7129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:56.651329 7129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:56.651465 7129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:56.660866 7129 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:56.660976 7129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:56.661154 7129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.69
I20240502 21:26:56.670689 7129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.69:33193
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.29.82.69
--webserver_port=40947
--tserver_master_addrs=127.29.82.126:37445
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.69
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:56.671912 7129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:56.673767 7129 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:56.686352 7138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:56.686944 7140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:56.687674 7137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:56.688354 7129 server_base.cc:1034] running on GCE node
I20240502 21:26:56.688952 7129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:56.689900 7129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:56.691128 7129 hybrid_clock.cc:648] HybridClock initialized: now 1714685216691078 us; error 39 us; skew 500 ppm
I20240502 21:26:56.691443 7129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:56.694157 7129 webserver.cc:460] Webserver started at http://127.29.82.69:40947/ using document root <none> and password file <none>
I20240502 21:26:56.694913 7129 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:56.695055 7129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:56.701050 7129 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 21:26:56.704025 7146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:56.705292 7129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 21:26:56.705437 7129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "39981a8dfe94462b9a73ba489af4c714"
format_stamp: "Formatted at 2024-05-02 21:26:23 on dist-test-slave-xzsm"
I20240502 21:26:56.706238 7129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:56.740404 7129 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:56.741328 7129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:56.741654 7129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:56.742621 7129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:56.744351 7129 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:56.744427 7129 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:56.744551 7129 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:56.744617 7129 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:56.794085 7129 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.69:33193
I20240502 21:26:56.794163 7260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.69:33193 every 8 connection(s)
I20240502 21:26:56.795785 7129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:26:56.800689 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 7129
I20240502 21:26:56.801362 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.70:40719
--local_ip_for_outbound_sockets=127.29.82.70
--tserver_master_addrs=127.29.82.126:37445
--webserver_port=34273
--webserver_interface=127.29.82.70
--builtin_ntp_servers=127.29.82.84:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:26:56.809252 7261 heartbeater.cc:347] Connected to a master server at 127.29.82.126:37445
I20240502 21:26:56.809577 7261 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:56.810370 7261 heartbeater.cc:511] Master 127.29.82.126:37445 requested a full tablet report, sending...
I20240502 21:26:56.811972 4292 ts_manager.cc:194] Re-registered known tserver with Master: 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:26:56.813055 4292 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.69:36131
W20240502 21:26:57.173321 7265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:57.173619 7265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:57.173774 7265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:57.182176 7265 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:57.182318 7265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:57.182510 7265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.70
I20240502 21:26:57.192631 7265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.70:40719
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.29.82.70
--webserver_port=34273
--tserver_master_addrs=127.29.82.126:37445
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.70
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:57.193987 7265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:57.195837 7265 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:57.209061 7271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:57.209559 7272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:57.210177 7274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:57.211202 7265 server_base.cc:1034] running on GCE node
I20240502 21:26:57.211894 7265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:57.212947 7265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:57.214149 7265 hybrid_clock.cc:648] HybridClock initialized: now 1714685217214081 us; error 62 us; skew 500 ppm
I20240502 21:26:57.214517 7265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:57.216964 7265 webserver.cc:460] Webserver started at http://127.29.82.70:34273/ using document root <none> and password file <none>
I20240502 21:26:57.217693 7265 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:57.217837 7265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:57.223664 7265 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20240502 21:26:57.226550 7280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:57.228098 7265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:57.228271 7265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "d0456c5e55a542128688904e003d403c"
format_stamp: "Formatted at 2024-05-02 21:26:23 on dist-test-slave-xzsm"
I20240502 21:26:57.229354 7265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:57.255564 7265 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:57.256541 7265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:57.256863 7265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:57.257862 7265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:57.259308 7265 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:57.259382 7265 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:57.259503 7265 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:57.259552 7265 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:57.309528 7265 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.70:40719
I20240502 21:26:57.309981 7394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.70:40719 every 8 connection(s)
I20240502 21:26:57.311465 7265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 21:26:57.316325 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 7265
I20240502 21:26:57.317248 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.71:33913
--local_ip_for_outbound_sockets=127.29.82.71
--tserver_master_addrs=127.29.82.126:37445
--webserver_port=46521
--webserver_interface=127.29.82.71
--builtin_ntp_servers=127.29.82.84:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:26:57.357304 7395 heartbeater.cc:347] Connected to a master server at 127.29.82.126:37445
I20240502 21:26:57.357686 7395 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:57.358609 7395 heartbeater.cc:511] Master 127.29.82.126:37445 requested a full tablet report, sending...
I20240502 21:26:57.360539 4292 ts_manager.cc:194] Re-registered known tserver with Master: d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:26:57.361481 4292 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.70:40413
W20240502 21:26:57.693737 7399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:57.694070 7399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:57.694226 7399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:57.702648 7399 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:57.702791 7399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:57.702953 7399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.71
I20240502 21:26:57.713241 7399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.71:33913
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.29.82.71
--webserver_port=46521
--tserver_master_addrs=127.29.82.126:37445
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.71
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:57.714639 7399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:57.716426 7399 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:57.730180 7406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:57.730760 7405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:57.731457 7409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:57.732270 7399 server_base.cc:1034] running on GCE node
I20240502 21:26:57.732841 7399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:57.733803 7399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:57.735038 7399 hybrid_clock.cc:648] HybridClock initialized: now 1714685217734943 us; error 49 us; skew 500 ppm
I20240502 21:26:57.735474 7399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:57.738184 7399 webserver.cc:460] Webserver started at http://127.29.82.71:46521/ using document root <none> and password file <none>
I20240502 21:26:57.738932 7399 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:57.739092 7399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:57.744997 7399 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20240502 21:26:57.748606 7415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:57.750092 7399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:57.750315 7399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "31a72286397c4edc9a63eca3412be168"
format_stamp: "Formatted at 2024-05-02 21:26:23 on dist-test-slave-xzsm"
I20240502 21:26:57.751353 7399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:57.784327 7399 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:57.785166 7399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:57.785511 7399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:57.786629 7399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:57.788976 7399 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:57.789070 7399 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:57.789201 7399 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:57.789266 7399 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:57.816007 7261 heartbeater.cc:503] Master 127.29.82.126:37445 was elected leader, sending a full tablet report...
I20240502 21:26:57.836894 7399 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.71:33913
I20240502 21:26:57.836974 7529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.71:33913 every 8 connection(s)
I20240502 21:26:57.838416 7399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 21:26:57.838918 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 7399
I20240502 21:26:57.839998 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.72:41223
--local_ip_for_outbound_sockets=127.29.82.72
--tserver_master_addrs=127.29.82.126:37445
--webserver_port=37911
--webserver_interface=127.29.82.72
--builtin_ntp_servers=127.29.82.84:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:26:57.853433 7530 heartbeater.cc:347] Connected to a master server at 127.29.82.126:37445
I20240502 21:26:57.853753 7530 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:57.854640 7530 heartbeater.cc:511] Master 127.29.82.126:37445 requested a full tablet report, sending...
I20240502 21:26:57.856161 4292 ts_manager.cc:194] Re-registered known tserver with Master: 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:26:57.857013 4292 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.71:54677
W20240502 21:26:58.202692 7533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:58.202961 7533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:58.203109 7533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:58.211012 7533 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:58.211136 7533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:58.211308 7533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.72
I20240502 21:26:58.222358 7533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.72:41223
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.29.82.72
--webserver_port=37911
--tserver_master_addrs=127.29.82.126:37445
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.72
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:58.223896 7533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:58.225979 7533 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:58.242398 7539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:58.243377 7540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:58.245041 7542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:58.245761 7533 server_base.cc:1034] running on GCE node
I20240502 21:26:58.246310 7533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:58.247241 7533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:58.248432 7533 hybrid_clock.cc:648] HybridClock initialized: now 1714685218248387 us; error 34 us; skew 500 ppm
I20240502 21:26:58.248802 7533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:58.252256 7533 webserver.cc:460] Webserver started at http://127.29.82.72:37911/ using document root <none> and password file <none>
I20240502 21:26:58.252976 7533 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:58.253104 7533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:58.258978 7533 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 21:26:58.261770 7548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:58.262990 7533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:58.263144 7533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "0ddd486109f5434cace13d60d633ac04"
format_stamp: "Formatted at 2024-05-02 21:26:24 on dist-test-slave-xzsm"
I20240502 21:26:58.264021 7533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:58.279124 7533 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:58.280223 7533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:58.280571 7533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:58.281491 7533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:58.282991 7533 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:58.283075 7533 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:58.283210 7533 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:58.283269 7533 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:58.352080 7533 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.72:41223
I20240502 21:26:58.352125 7662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.72:41223 every 8 connection(s)
I20240502 21:26:58.353837 7533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 21:26:58.356928 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 7533
I20240502 21:26:58.364336 7395 heartbeater.cc:503] Master 127.29.82.126:37445 was elected leader, sending a full tablet report...
I20240502 21:26:58.370185 7663 heartbeater.cc:347] Connected to a master server at 127.29.82.126:37445
I20240502 21:26:58.370545 7663 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:58.371964 7663 heartbeater.cc:511] Master 127.29.82.126:37445 requested a full tablet report, sending...
I20240502 21:26:58.373539 4292 ts_manager.cc:194] Re-registered known tserver with Master: 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:26:58.374480 4292 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.72:58139
I20240502 21:26:58.375871 4292 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:26:58.387710 4422 tablet_service.cc:1467] Processing CreateTablet for tablet f123d9ce64344c5082d85163be301c44 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=10b2973fab9d47049e186a9b5330a6c8]), partition=
I20240502 21:26:58.388182 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f123d9ce64344c5082d85163be301c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.428571 7195 tablet_service.cc:1467] Processing CreateTablet for tablet f123d9ce64344c5082d85163be301c44 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=10b2973fab9d47049e186a9b5330a6c8]), partition=
I20240502 21:26:58.429742 7195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f123d9ce64344c5082d85163be301c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.430881 7681 tablet_bootstrap.cc:492] T f123d9ce64344c5082d85163be301c44 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:26:58.432955 7681 tablet_bootstrap.cc:654] T f123d9ce64344c5082d85163be301c44 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:58.450587 7681 tablet_bootstrap.cc:492] T f123d9ce64344c5082d85163be301c44 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:26:58.450945 7681 ts_tablet_manager.cc:1367] T f123d9ce64344c5082d85163be301c44 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.006s
I20240502 21:26:58.451956 7681 raft_consensus.cc:348] T f123d9ce64344c5082d85163be301c44 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:26:58.452337 7681 raft_consensus.cc:374] T f123d9ce64344c5082d85163be301c44 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:58.452486 7681 raft_consensus.cc:729] T f123d9ce64344c5082d85163be301c44 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:26:58.452834 7681 consensus_queue.cc:260] T f123d9ce64344c5082d85163be301c44 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:26:58.453492 7681 ts_tablet_manager.cc:1398] T f123d9ce64344c5082d85163be301c44 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:58.453764 7464 tablet_service.cc:1467] Processing CreateTablet for tablet f123d9ce64344c5082d85163be301c44 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=10b2973fab9d47049e186a9b5330a6c8]), partition=
I20240502 21:26:58.454777 7464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f123d9ce64344c5082d85163be301c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.456864 7683 tablet_bootstrap.cc:492] T f123d9ce64344c5082d85163be301c44 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:26:58.460227 7683 tablet_bootstrap.cc:654] T f123d9ce64344c5082d85163be301c44 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:58.461527 7683 log.cc:824] T f123d9ce64344c5082d85163be301c44 P 39981a8dfe94462b9a73ba489af4c714: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:58.470432 7683 tablet_bootstrap.cc:492] T f123d9ce64344c5082d85163be301c44 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:26:58.470563 7686 tablet_bootstrap.cc:492] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:26:58.470784 7683 ts_tablet_manager.cc:1367] T f123d9ce64344c5082d85163be301c44 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.008s
I20240502 21:26:58.474367 7686 tablet_bootstrap.cc:654] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:58.475391 7686 log.cc:824] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:58.477734 7686 tablet_bootstrap.cc:492] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:26:58.477994 7686 ts_tablet_manager.cc:1367] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20240502 21:26:58.479926 7683 raft_consensus.cc:348] T f123d9ce64344c5082d85163be301c44 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:26:58.480448 7683 raft_consensus.cc:374] T f123d9ce64344c5082d85163be301c44 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:58.480592 7683 raft_consensus.cc:729] T f123d9ce64344c5082d85163be301c44 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:26:58.481470 7683 consensus_queue.cc:260] T f123d9ce64344c5082d85163be301c44 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:26:58.485442 7683 ts_tablet_manager.cc:1398] T f123d9ce64344c5082d85163be301c44 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.014s user 0.010s sys 0.002s
I20240502 21:26:58.490314 7686 raft_consensus.cc:348] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:26:58.490694 7686 raft_consensus.cc:374] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:58.490794 7686 raft_consensus.cc:729] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:26:58.491420 7686 consensus_queue.cc:260] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:26:58.492623 7530 heartbeater.cc:503] Master 127.29.82.126:37445 was elected leader, sending a full tablet report...
I20240502 21:26:58.493155 7686 ts_tablet_manager.cc:1398] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.015s user 0.009s sys 0.003s
I20240502 21:26:58.504657 7689 raft_consensus.cc:482] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:58.504942 7689 raft_consensus.cc:504] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:26:58.506637 7689 leader_election.cc:288] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:58.533071 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f123d9ce64344c5082d85163be301c44" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:26:58.533587 7215 raft_consensus.cc:2454] T f123d9ce64344c5082d85163be301c44 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:26:58.534297 7419 leader_election.cc:302] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:26:58.535161 7689 raft_consensus.cc:2781] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:58.535383 7689 raft_consensus.cc:482] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:58.535491 7689 raft_consensus.cc:3037] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:58.537711 7689 raft_consensus.cc:504] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:26:58.538300 7689 leader_election.cc:288] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Requested vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:26:58.538933 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f123d9ce64344c5082d85163be301c44" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:26:58.539144 7215 raft_consensus.cc:3037] T f123d9ce64344c5082d85163be301c44 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:58.542348 7215 raft_consensus.cc:2454] T f123d9ce64344c5082d85163be301c44 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:26:58.542670 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f123d9ce64344c5082d85163be301c44" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:26:58.542812 7419 leader_election.cc:302] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:26:58.542891 4443 raft_consensus.cc:2454] T f123d9ce64344c5082d85163be301c44 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:26:58.543473 7689 raft_consensus.cc:2781] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:58.544031 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f123d9ce64344c5082d85163be301c44" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:26:58.544245 4440 raft_consensus.cc:3037] T f123d9ce64344c5082d85163be301c44 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:58.546846 4440 raft_consensus.cc:2454] T f123d9ce64344c5082d85163be301c44 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:26:58.547199 7689 raft_consensus.cc:686] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Becoming Leader. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Running, Role: LEADER
I20240502 21:26:58.547865 7689 consensus_queue.cc:237] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
W20240502 21:26:58.549026 7262 tablet.cc:2369] T f123d9ce64344c5082d85163be301c44 P 39981a8dfe94462b9a73ba489af4c714: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:58.553369 4295 catalog_manager.cc:5484] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a72286397c4edc9a63eca3412be168 (127.29.82.71). New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
W20240502 21:26:58.592235 7531 tablet.cc:2369] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:58.678296 4295 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:26:58.689445 7215 raft_consensus.cc:1261] T f123d9ce64344c5082d85163be301c44 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:58.690449 4440 raft_consensus.cc:1261] T f123d9ce64344c5082d85163be301c44 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:58.690621 7693 consensus_queue.cc:1035] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:58.691829 7693 consensus_queue.cc:1035] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:58.709982 7695 mvcc.cc:205] Tried to move back new op lower bound from 7023350655737339904 to 7023350655177895936. Current Snapshot: MvccSnapshot[applied={T|T < 7023350655737339904}]
I20240502 21:26:58.712841 7701 tablet.cc:1722] T f123d9ce64344c5082d85163be301c44 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:58.714747 7701 tablet.cc:1620] T f123d9ce64344c5082d85163be301c44 P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:26:58.708292 4295 catalog_manager.cc:5576] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193) does not have the latest schema for tablet f123d9ce64344c5082d85163be301c44 (table rebalancer_dynamic_table [id=10b2973fab9d47049e186a9b5330a6c8]). Expected version 1 got 0
I20240502 21:26:58.717123 4298 catalog_manager.cc:5576] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781) does not have the latest schema for tablet f123d9ce64344c5082d85163be301c44 (table rebalancer_dynamic_table [id=10b2973fab9d47049e186a9b5330a6c8]). Expected version 1 got 0
I20240502 21:26:58.717669 7702 tablet.cc:1722] T f123d9ce64344c5082d85163be301c44 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:58.721007 7702 tablet.cc:1620] T f123d9ce64344c5082d85163be301c44 P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:26:58.724814 4298 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=10b2973fab9d47049e186a9b5330a6c8] alter complete (version 1)
I20240502 21:26:58.733485 7700 tablet.cc:1722] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:58.737941 7700 tablet.cc:1620] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:26:58.835291 4295 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:26:58.845352 7700 tablet.cc:1722] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:58.845872 7700 tablet.cc:1620] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:26:58.845873 7702 tablet.cc:1722] T f123d9ce64344c5082d85163be301c44 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:58.846338 7702 tablet.cc:1620] T f123d9ce64344c5082d85163be301c44 P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:26:58.850484 7701 tablet.cc:1722] T f123d9ce64344c5082d85163be301c44 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:58.851038 7701 tablet.cc:1620] T f123d9ce64344c5082d85163be301c44 P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:26:58.856153 4295 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=10b2973fab9d47049e186a9b5330a6c8] alter complete (version 2)
I20240502 21:26:58.859843 4295 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240502 21:26:58.860179 4295 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240502 21:26:58.860074 4298 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:58.866180 4295 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 25ce93e8577248a4b915265fb781faae
I20240502 21:26:58.866951 4295 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 7feb601648e14a998cfc7331a6934ac1
I20240502 21:26:58.867357 4295 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 23c09be2916844a9b9bb5855d3566b86
I20240502 21:26:58.867334 4422 tablet_service.cc:1514] Processing DeleteTablet for tablet 25ce93e8577248a4b915265fb781faae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:58 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:26:58.868482 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet 7feb601648e14a998cfc7331a6934ac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:58 UTC) from {username='slave'} at 127.0.0.1:60018
I20240502 21:26:58.868546 4553 tablet_service.cc:1514] Processing DeleteTablet for tablet 23c09be2916844a9b9bb5855d3566b86 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:58 UTC) from {username='slave'} at 127.0.0.1:60018
I20240502 21:26:58.869828 4816 tablet_service.cc:1514] Processing DeleteTablet for tablet 25ce93e8577248a4b915265fb781faae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:58 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:26:58.870566 4818 tablet_service.cc:1514] Processing DeleteTablet for tablet 23c09be2916844a9b9bb5855d3566b86 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:58 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:26:58.871541 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet 25ce93e8577248a4b915265fb781faae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:58 UTC) from {username='slave'} at 127.0.0.1:44764
I20240502 21:26:58.872100 4685 tablet_service.cc:1514] Processing DeleteTablet for tablet 7feb601648e14a998cfc7331a6934ac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:58 UTC) from {username='slave'} at 127.0.0.1:44764
I20240502 21:26:58.878485 7464 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3be8c5eaac423db11b282739cd1d40 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=ab7a7a9b6e69455484db6423f42c7c06]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:58.878487 7195 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3be8c5eaac423db11b282739cd1d40 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=ab7a7a9b6e69455484db6423f42c7c06]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:58.879123 7195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3be8c5eaac423db11b282739cd1d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.879218 7464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3be8c5eaac423db11b282739cd1d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.879318 4553 tablet_service.cc:1467] Processing CreateTablet for tablet e8f42cdee5c24db982392d2057b588a2 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=ab7a7a9b6e69455484db6423f42c7c06]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:58.879900 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8f42cdee5c24db982392d2057b588a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.887544 7194 tablet_service.cc:1467] Processing CreateTablet for tablet e8f42cdee5c24db982392d2057b588a2 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=ab7a7a9b6e69455484db6423f42c7c06]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:58.888221 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8f42cdee5c24db982392d2057b588a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.892568 7686 tablet_bootstrap.cc:492] T 1d3be8c5eaac423db11b282739cd1d40 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:26:58.894641 7686 tablet_bootstrap.cc:654] T 1d3be8c5eaac423db11b282739cd1d40 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:58.896476 7683 tablet_bootstrap.cc:492] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:26:58.869112 7716 tablet_replica.cc:330] T 25ce93e8577248a4b915265fb781faae P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:26:58.870577 4422 tablet_service.cc:1514] Processing DeleteTablet for tablet 7feb601648e14a998cfc7331a6934ac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:58 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:26:58.870709 4423 tablet_service.cc:1514] Processing DeleteTablet for tablet 23c09be2916844a9b9bb5855d3566b86 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:58 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:26:58.898078 7716 raft_consensus.cc:2229] T 25ce93e8577248a4b915265fb781faae P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:58.898572 7683 tablet_bootstrap.cc:654] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:58.899616 7716 raft_consensus.cc:2258] T 25ce93e8577248a4b915265fb781faae P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:58.903254 7716 ts_tablet_manager.cc:1875] T 25ce93e8577248a4b915265fb781faae P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:58.904148 7719 tablet_replica.cc:330] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:26:58.906783 7719 raft_consensus.cc:2229] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:58.909366 7719 raft_consensus.cc:2258] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:58.912829 7686 tablet_bootstrap.cc:492] T 1d3be8c5eaac423db11b282739cd1d40 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:26:58.913079 7686 ts_tablet_manager.cc:1367] T 1d3be8c5eaac423db11b282739cd1d40 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.003s
I20240502 21:26:58.914302 7686 raft_consensus.cc:348] T 1d3be8c5eaac423db11b282739cd1d40 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:26:58.914475 7718 tablet_replica.cc:330] T 25ce93e8577248a4b915265fb781faae P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:26:58.914664 7686 raft_consensus.cc:374] T 1d3be8c5eaac423db11b282739cd1d40 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:58.914781 7686 raft_consensus.cc:729] T 1d3be8c5eaac423db11b282739cd1d40 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:26:58.916080 7719 ts_tablet_manager.cc:1875] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:58.916662 7718 raft_consensus.cc:2229] T 25ce93e8577248a4b915265fb781faae P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:58.917493 7686 consensus_queue.cc:260] T 1d3be8c5eaac423db11b282739cd1d40 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:26:58.918092 7686 ts_tablet_manager.cc:1398] T 1d3be8c5eaac423db11b282739cd1d40 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 21:26:58.919056 7718 raft_consensus.cc:2258] T 25ce93e8577248a4b915265fb781faae P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:58.923763 7718 ts_tablet_manager.cc:1875] T 25ce93e8577248a4b915265fb781faae P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:58.925952 7195 tablet_service.cc:1467] Processing CreateTablet for tablet af97ce1401784f45807466e2ebb970f7 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=ab7a7a9b6e69455484db6423f42c7c06]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:58.926550 7195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af97ce1401784f45807466e2ebb970f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.925608 4816 tablet_service.cc:1467] Processing CreateTablet for tablet af97ce1401784f45807466e2ebb970f7 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=ab7a7a9b6e69455484db6423f42c7c06]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:58.927102 4816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af97ce1401784f45807466e2ebb970f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.927386 7464 tablet_service.cc:1467] Processing CreateTablet for tablet af97ce1401784f45807466e2ebb970f7 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=ab7a7a9b6e69455484db6423f42c7c06]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:58.927826 7464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af97ce1401784f45807466e2ebb970f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.940618 7719 ts_tablet_manager.cc:1888] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:58.946723 7725 tablet_bootstrap.cc:492] T e8f42cdee5c24db982392d2057b588a2 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:26:58.949106 7725 tablet_bootstrap.cc:654] T e8f42cdee5c24db982392d2057b588a2 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:58.950331 7719 log.cc:1196] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/25ce93e8577248a4b915265fb781faae
I20240502 21:26:58.951251 7719 ts_tablet_manager.cc:1909] T 25ce93e8577248a4b915265fb781faae P 398c3c8f3195492c900439ad5e1931bb: Deleting consensus metadata
I20240502 21:26:58.951894 7686 tablet_bootstrap.cc:492] T af97ce1401784f45807466e2ebb970f7 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:26:58.952816 7716 ts_tablet_manager.cc:1888] T 25ce93e8577248a4b915265fb781faae P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:58.953006 7716 log.cc:1196] T 25ce93e8577248a4b915265fb781faae P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/25ce93e8577248a4b915265fb781faae
I20240502 21:26:58.953562 7716 ts_tablet_manager.cc:1909] T 25ce93e8577248a4b915265fb781faae P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:26:58.953989 7686 tablet_bootstrap.cc:654] T af97ce1401784f45807466e2ebb970f7 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:58.957355 7686 tablet_bootstrap.cc:492] T af97ce1401784f45807466e2ebb970f7 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:26:58.957595 7686 ts_tablet_manager.cc:1367] T af97ce1401784f45807466e2ebb970f7 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240502 21:26:58.958472 7686 raft_consensus.cc:348] T af97ce1401784f45807466e2ebb970f7 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:26:58.958784 7686 raft_consensus.cc:374] T af97ce1401784f45807466e2ebb970f7 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:58.958922 7686 raft_consensus.cc:729] T af97ce1401784f45807466e2ebb970f7 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:26:58.959388 7686 consensus_queue.cc:260] T af97ce1401784f45807466e2ebb970f7 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:26:58.960595 4298 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:26:58.961315 7725 tablet_bootstrap.cc:492] T e8f42cdee5c24db982392d2057b588a2 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:26:58.961453 7725 ts_tablet_manager.cc:1367] T e8f42cdee5c24db982392d2057b588a2 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.002s
I20240502 21:26:58.962211 7725 raft_consensus.cc:348] T e8f42cdee5c24db982392d2057b588a2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:26:58.962800 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet 25ce93e8577248a4b915265fb781faae (table rebalance_test_table_0 [id=927c4d77ce3e435da867fbd206041c89]) successfully deleted
I20240502 21:26:58.963722 7724 tablet_bootstrap.cc:492] T af97ce1401784f45807466e2ebb970f7 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:26:58.964965 7716 tablet_replica.cc:330] T 7feb601648e14a998cfc7331a6934ac1 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:26:58.965376 7716 raft_consensus.cc:2229] T 7feb601648e14a998cfc7331a6934ac1 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:58.965641 7725 raft_consensus.cc:374] T e8f42cdee5c24db982392d2057b588a2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:58.965765 7725 raft_consensus.cc:729] T e8f42cdee5c24db982392d2057b588a2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:26:58.966019 7716 raft_consensus.cc:2258] T 7feb601648e14a998cfc7331a6934ac1 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:58.966027 7725 consensus_queue.cc:260] T e8f42cdee5c24db982392d2057b588a2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:26:58.966111 7328 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3be8c5eaac423db11b282739cd1d40 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=ab7a7a9b6e69455484db6423f42c7c06]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:58.968029 7725 ts_tablet_manager.cc:1398] T e8f42cdee5c24db982392d2057b588a2 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20240502 21:26:58.969010 7686 ts_tablet_manager.cc:1398] T af97ce1401784f45807466e2ebb970f7 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 21:26:58.972530 7718 ts_tablet_manager.cc:1888] T 25ce93e8577248a4b915265fb781faae P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:58.972751 7718 log.cc:1196] T 25ce93e8577248a4b915265fb781faae P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/25ce93e8577248a4b915265fb781faae
I20240502 21:26:58.972780 7700 tablet.cc:1722] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:58.974020 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet 25ce93e8577248a4b915265fb781faae (table rebalance_test_table_0 [id=927c4d77ce3e435da867fbd206041c89]) successfully deleted
I20240502 21:26:58.974282 7718 ts_tablet_manager.cc:1909] T 25ce93e8577248a4b915265fb781faae P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:26:58.974380 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3be8c5eaac423db11b282739cd1d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.975951 7718 tablet_replica.cc:330] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:26:58.976190 7718 raft_consensus.cc:2229] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:58.976997 7719 tablet_replica.cc:330] T 7feb601648e14a998cfc7331a6934ac1 P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:26:58.977336 7719 raft_consensus.cc:2229] T 7feb601648e14a998cfc7331a6934ac1 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:58.977686 7719 raft_consensus.cc:2258] T 7feb601648e14a998cfc7331a6934ac1 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:58.978559 7719 ts_tablet_manager.cc:1875] T 7feb601648e14a998cfc7331a6934ac1 P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:58.980660 7718 raft_consensus.cc:2258] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:58.981189 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet 25ce93e8577248a4b915265fb781faae (table rebalance_test_table_0 [id=927c4d77ce3e435da867fbd206041c89]) successfully deleted
I20240502 21:26:58.983110 7718 ts_tablet_manager.cc:1875] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:58.983709 7702 tablet.cc:1722] T f123d9ce64344c5082d85163be301c44 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:58.991466 7719 ts_tablet_manager.cc:1888] T 7feb601648e14a998cfc7331a6934ac1 P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:58.995163 7719 log.cc:1196] T 7feb601648e14a998cfc7331a6934ac1 P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/7feb601648e14a998cfc7331a6934ac1
I20240502 21:26:58.995779 7719 ts_tablet_manager.cc:1909] T 7feb601648e14a998cfc7331a6934ac1 P 398c3c8f3195492c900439ad5e1931bb: Deleting consensus metadata
I20240502 21:26:58.996016 7683 tablet_bootstrap.cc:492] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:26:58.996296 7683 ts_tablet_manager.cc:1367] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.100s user 0.001s sys 0.011s
I20240502 21:26:58.997037 7718 ts_tablet_manager.cc:1888] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:58.997287 7718 log.cc:1196] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/23c09be2916844a9b9bb5855d3566b86
I20240502 21:26:58.997278 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet 7feb601648e14a998cfc7331a6934ac1 (table rebalance_test_table_0 [id=927c4d77ce3e435da867fbd206041c89]) successfully deleted
I20240502 21:26:58.997807 7718 ts_tablet_manager.cc:1909] T 23c09be2916844a9b9bb5855d3566b86 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:26:58.998539 7683 raft_consensus.cc:348] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:26:59.000309 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet 23c09be2916844a9b9bb5855d3566b86 (table rebalance_test_table_0 [id=927c4d77ce3e435da867fbd206041c89]) successfully deleted
I20240502 21:26:59.001564 4292 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=10b2973fab9d47049e186a9b5330a6c8] alter complete (version 3)
I20240502 21:26:59.001631 7717 tablet_replica.cc:330] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:26:59.002419 7717 raft_consensus.cc:2229] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:59.003454 7717 raft_consensus.cc:2258] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.003998 7683 raft_consensus.cc:374] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.005482 7683 raft_consensus.cc:729] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.005905 7683 consensus_queue.cc:260] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:26:59.011922 7683 ts_tablet_manager.cc:1398] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.015s user 0.001s sys 0.002s
I20240502 21:26:59.012334 7683 tablet_bootstrap.cc:492] T e8f42cdee5c24db982392d2057b588a2 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:26:59.017585 7724 tablet_bootstrap.cc:654] T af97ce1401784f45807466e2ebb970f7 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:58.977833 7701 tablet.cc:1722] T f123d9ce64344c5082d85163be301c44 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:59.018926 7716 ts_tablet_manager.cc:1875] T 7feb601648e14a998cfc7331a6934ac1 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:59.027508 7717 ts_tablet_manager.cc:1875] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:59.032377 7716 ts_tablet_manager.cc:1888] T 7feb601648e14a998cfc7331a6934ac1 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:59.032570 7716 log.cc:1196] T 7feb601648e14a998cfc7331a6934ac1 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/7feb601648e14a998cfc7331a6934ac1
I20240502 21:26:59.033013 7716 ts_tablet_manager.cc:1909] T 7feb601648e14a998cfc7331a6934ac1 P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:26:59.034005 7716 tablet_replica.cc:330] T 23c09be2916844a9b9bb5855d3566b86 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:26:59.034097 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet 7feb601648e14a998cfc7331a6934ac1 (table rebalance_test_table_0 [id=927c4d77ce3e435da867fbd206041c89]) successfully deleted
I20240502 21:26:59.035352 7683 tablet_bootstrap.cc:654] T e8f42cdee5c24db982392d2057b588a2 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.035104 7597 tablet_service.cc:1467] Processing CreateTablet for tablet e8f42cdee5c24db982392d2057b588a2 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=ab7a7a9b6e69455484db6423f42c7c06]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:59.036600 7716 raft_consensus.cc:2229] T 23c09be2916844a9b9bb5855d3566b86 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:59.036695 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8f42cdee5c24db982392d2057b588a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.036854 7716 raft_consensus.cc:2258] T 23c09be2916844a9b9bb5855d3566b86 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.038807 7716 ts_tablet_manager.cc:1875] T 23c09be2916844a9b9bb5855d3566b86 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:59.054354 7716 ts_tablet_manager.cc:1888] T 23c09be2916844a9b9bb5855d3566b86 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:59.054529 7716 log.cc:1196] T 23c09be2916844a9b9bb5855d3566b86 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/23c09be2916844a9b9bb5855d3566b86
I20240502 21:26:59.055074 7716 ts_tablet_manager.cc:1909] T 23c09be2916844a9b9bb5855d3566b86 P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:26:59.056299 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet 23c09be2916844a9b9bb5855d3566b86 (table rebalance_test_table_0 [id=927c4d77ce3e435da867fbd206041c89]) successfully deleted
I20240502 21:26:59.058598 7732 tablet_bootstrap.cc:492] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:26:59.062345 7732 tablet_bootstrap.cc:654] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.063727 7732 log.cc:824] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:59.065189 7683 tablet_bootstrap.cc:492] T e8f42cdee5c24db982392d2057b588a2 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:26:59.067760 7683 ts_tablet_manager.cc:1367] T e8f42cdee5c24db982392d2057b588a2 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.055s user 0.005s sys 0.001s
I20240502 21:26:59.068140 7732 tablet_bootstrap.cc:492] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:26:59.068473 7732 ts_tablet_manager.cc:1367] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20240502 21:26:59.069114 7724 tablet_bootstrap.cc:492] T af97ce1401784f45807466e2ebb970f7 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:26:59.069329 7724 ts_tablet_manager.cc:1367] T af97ce1401784f45807466e2ebb970f7 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.106s user 0.015s sys 0.012s
I20240502 21:26:59.070328 7683 raft_consensus.cc:348] T e8f42cdee5c24db982392d2057b588a2 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:26:59.070621 7683 raft_consensus.cc:374] T e8f42cdee5c24db982392d2057b588a2 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.070717 7683 raft_consensus.cc:729] T e8f42cdee5c24db982392d2057b588a2 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.070495 7724 raft_consensus.cc:348] T af97ce1401784f45807466e2ebb970f7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:26:59.070937 7724 raft_consensus.cc:374] T af97ce1401784f45807466e2ebb970f7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.071071 7724 raft_consensus.cc:729] T af97ce1401784f45807466e2ebb970f7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.071404 7724 consensus_queue.cc:260] T af97ce1401784f45807466e2ebb970f7 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:26:59.072208 7724 ts_tablet_manager.cc:1398] T af97ce1401784f45807466e2ebb970f7 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:59.077509 7732 raft_consensus.cc:348] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:26:59.077992 7732 raft_consensus.cc:374] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.078135 7732 raft_consensus.cc:729] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.078600 7734 tablet_bootstrap.cc:492] T 1d3be8c5eaac423db11b282739cd1d40 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:26:59.078981 7732 consensus_queue.cc:260] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:26:59.077617 7683 consensus_queue.cc:260] T e8f42cdee5c24db982392d2057b588a2 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:26:59.082878 7734 tablet_bootstrap.cc:654] T 1d3be8c5eaac423db11b282739cd1d40 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.084888 7734 log.cc:824] T 1d3be8c5eaac423db11b282739cd1d40 P d0456c5e55a542128688904e003d403c: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:59.085700 7683 ts_tablet_manager.cc:1398] T e8f42cdee5c24db982392d2057b588a2 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20240502 21:26:59.086205 7663 heartbeater.cc:503] Master 127.29.82.126:37445 was elected leader, sending a full tablet report...
I20240502 21:26:59.086767 7717 ts_tablet_manager.cc:1888] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:59.086936 7717 log.cc:1196] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/7feb601648e14a998cfc7331a6934ac1
I20240502 21:26:59.087265 7732 ts_tablet_manager.cc:1398] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.019s user 0.012s sys 0.005s
I20240502 21:26:59.088388 7683 tablet_bootstrap.cc:492] T af97ce1401784f45807466e2ebb970f7 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:26:59.093401 7683 tablet_bootstrap.cc:654] T af97ce1401784f45807466e2ebb970f7 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.094879 7717 ts_tablet_manager.cc:1909] T 7feb601648e14a998cfc7331a6934ac1 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting consensus metadata
I20240502 21:26:59.097648 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet 7feb601648e14a998cfc7331a6934ac1 (table rebalance_test_table_0 [id=927c4d77ce3e435da867fbd206041c89]) successfully deleted
I20240502 21:26:59.101962 7717 tablet_replica.cc:330] T 23c09be2916844a9b9bb5855d3566b86 P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:26:59.102312 7717 raft_consensus.cc:2229] T 23c09be2916844a9b9bb5855d3566b86 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:59.103137 7717 raft_consensus.cc:2258] T 23c09be2916844a9b9bb5855d3566b86 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.103287 7734 tablet_bootstrap.cc:492] T 1d3be8c5eaac423db11b282739cd1d40 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:26:59.103626 7734 ts_tablet_manager.cc:1367] T 1d3be8c5eaac423db11b282739cd1d40 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.000s
I20240502 21:26:59.104262 7717 ts_tablet_manager.cc:1875] T 23c09be2916844a9b9bb5855d3566b86 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 21:26:59.105850 7664 tablet.cc:2369] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:59.107875 7683 tablet_bootstrap.cc:492] T af97ce1401784f45807466e2ebb970f7 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:26:59.108120 7683 ts_tablet_manager.cc:1367] T af97ce1401784f45807466e2ebb970f7 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.002s
I20240502 21:26:59.108970 7683 raft_consensus.cc:348] T af97ce1401784f45807466e2ebb970f7 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:26:59.109265 7683 raft_consensus.cc:374] T af97ce1401784f45807466e2ebb970f7 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.109397 7683 raft_consensus.cc:729] T af97ce1401784f45807466e2ebb970f7 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.109743 7683 consensus_queue.cc:260] T af97ce1401784f45807466e2ebb970f7 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:26:59.111176 7683 ts_tablet_manager.cc:1398] T af97ce1401784f45807466e2ebb970f7 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:59.113363 7734 raft_consensus.cc:348] T 1d3be8c5eaac423db11b282739cd1d40 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:26:59.113840 7734 raft_consensus.cc:374] T 1d3be8c5eaac423db11b282739cd1d40 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.113962 7734 raft_consensus.cc:729] T 1d3be8c5eaac423db11b282739cd1d40 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.114095 4292 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:59.114317 4292 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:59.114933 7734 consensus_queue.cc:260] T 1d3be8c5eaac423db11b282739cd1d40 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:26:59.118659 7717 ts_tablet_manager.cc:1888] T 23c09be2916844a9b9bb5855d3566b86 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:59.118868 7717 log.cc:1196] T 23c09be2916844a9b9bb5855d3566b86 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/23c09be2916844a9b9bb5855d3566b86
I20240502 21:26:59.119369 7717 ts_tablet_manager.cc:1909] T 23c09be2916844a9b9bb5855d3566b86 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting consensus metadata
I20240502 21:26:59.120703 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet 23c09be2916844a9b9bb5855d3566b86 (table rebalance_test_table_0 [id=927c4d77ce3e435da867fbd206041c89]) successfully deleted
I20240502 21:26:59.121068 4292 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet f123d9ce64344c5082d85163be301c44
I20240502 21:26:59.122138 4423 tablet_service.cc:1514] Processing DeleteTablet for tablet f123d9ce64344c5082d85163be301c44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:59 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:26:59.122107 7194 tablet_service.cc:1514] Processing DeleteTablet for tablet f123d9ce64344c5082d85163be301c44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:59 UTC) from {username='slave'} at 127.0.0.1:44652
I20240502 21:26:59.122452 7716 tablet_replica.cc:330] T f123d9ce64344c5082d85163be301c44 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:26:59.122712 7716 raft_consensus.cc:2229] T f123d9ce64344c5082d85163be301c44 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:59.122968 7716 raft_consensus.cc:2258] T f123d9ce64344c5082d85163be301c44 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.123471 7464 tablet_service.cc:1514] Processing DeleteTablet for tablet f123d9ce64344c5082d85163be301c44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:59 UTC) from {username='slave'} at 127.0.0.1:37182
I20240502 21:26:59.124073 7716 ts_tablet_manager.cc:1875] T f123d9ce64344c5082d85163be301c44 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 21:26:59.127199 4885 tablet.cc:2369] T af97ce1401784f45807466e2ebb970f7 P 73a1f2ca1f544de9b0e3b37015f85328: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:59.131525 7716 ts_tablet_manager.cc:1888] T f123d9ce64344c5082d85163be301c44 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:59.131716 7716 log.cc:1196] T f123d9ce64344c5082d85163be301c44 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/f123d9ce64344c5082d85163be301c44
I20240502 21:26:59.132237 7716 ts_tablet_manager.cc:1909] T f123d9ce64344c5082d85163be301c44 P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:26:59.133550 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet f123d9ce64344c5082d85163be301c44 (table rebalancer_dynamic_table_new_name [id=10b2973fab9d47049e186a9b5330a6c8]) successfully deleted
I20240502 21:26:59.138370 7734 ts_tablet_manager.cc:1398] T 1d3be8c5eaac423db11b282739cd1d40 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.035s user 0.011s sys 0.006s
I20240502 21:26:59.141708 7740 tablet_replica.cc:330] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:26:59.142199 7740 raft_consensus.cc:2229] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:59.142853 7740 raft_consensus.cc:2258] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.145887 7740 ts_tablet_manager.cc:1875] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:59.146787 7739 tablet_replica.cc:330] T f123d9ce64344c5082d85163be301c44 P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:26:59.147394 7739 raft_consensus.cc:2229] T f123d9ce64344c5082d85163be301c44 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:59.148262 7739 raft_consensus.cc:2258] T f123d9ce64344c5082d85163be301c44 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.150154 7739 ts_tablet_manager.cc:1875] T f123d9ce64344c5082d85163be301c44 P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:59.155613 7740 ts_tablet_manager.cc:1888] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:59.155792 7740 log.cc:1196] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/f123d9ce64344c5082d85163be301c44
I20240502 21:26:59.156322 7740 ts_tablet_manager.cc:1909] T f123d9ce64344c5082d85163be301c44 P 31a72286397c4edc9a63eca3412be168: Deleting consensus metadata
I20240502 21:26:59.158144 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet f123d9ce64344c5082d85163be301c44 (table rebalancer_dynamic_table_new_name [id=10b2973fab9d47049e186a9b5330a6c8]) successfully deleted
I20240502 21:26:59.159137 7739 ts_tablet_manager.cc:1888] T f123d9ce64344c5082d85163be301c44 P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:59.159282 7739 log.cc:1196] T f123d9ce64344c5082d85163be301c44 P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/f123d9ce64344c5082d85163be301c44
I20240502 21:26:59.159917 7739 ts_tablet_manager.cc:1909] T f123d9ce64344c5082d85163be301c44 P 39981a8dfe94462b9a73ba489af4c714: Deleting consensus metadata
I20240502 21:26:59.161278 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet f123d9ce64344c5082d85163be301c44 (table rebalancer_dynamic_table_new_name [id=10b2973fab9d47049e186a9b5330a6c8]) successfully deleted
I20240502 21:26:59.167093 7735 raft_consensus.cc:482] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:59.167379 7735 raft_consensus.cc:504] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:26:59.169624 7735 leader_election.cc:288] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:26:59.180629 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8f42cdee5c24db982392d2057b588a2" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:26:59.180933 4574 raft_consensus.cc:2454] T e8f42cdee5c24db982392d2057b588a2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:26:59.181668 7552 leader_election.cc:302] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:59.182174 7735 raft_consensus.cc:2781] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:59.182379 7735 raft_consensus.cc:482] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:59.182493 7735 raft_consensus.cc:3037] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.186322 7735 raft_consensus.cc:504] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:26:59.188175 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8f42cdee5c24db982392d2057b588a2" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:26:59.188385 4574 raft_consensus.cc:3037] T e8f42cdee5c24db982392d2057b588a2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.188990 7735 leader_election.cc:288] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Requested vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:26:59.191208 4574 raft_consensus.cc:2454] T e8f42cdee5c24db982392d2057b588a2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:26:59.191628 7552 leader_election.cc:302] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:26:59.191939 7735 raft_consensus.cc:2781] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:59.192299 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8f42cdee5c24db982392d2057b588a2" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:26:59.192536 7215 raft_consensus.cc:2454] T e8f42cdee5c24db982392d2057b588a2 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:26:59.192538 7735 raft_consensus.cc:686] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Becoming Leader. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Running, Role: LEADER
I20240502 21:26:59.193106 7735 consensus_queue.cc:237] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:26:59.193652 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8f42cdee5c24db982392d2057b588a2" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:26:59.193868 7215 raft_consensus.cc:3037] T e8f42cdee5c24db982392d2057b588a2 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.197264 7215 raft_consensus.cc:2454] T e8f42cdee5c24db982392d2057b588a2 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
W20240502 21:26:59.197640 4622 tablet.cc:2369] T e8f42cdee5c24db982392d2057b588a2 P 7a736a8ebb164ce0a051c0fccd76c3b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:59.200345 4292 catalog_manager.cc:5484] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ddd486109f5434cace13d60d633ac04 (127.29.82.72). New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:26:59.225001 4292 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:26:59.241405 7597 tablet_service.cc:1467] Processing CreateTablet for tablet 319b287b9c05456882890f0bb4cd1d35 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7a9b82ace20943c99c2d9d15fd3a73c8]), partition=
I20240502 21:26:59.241801 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 319b287b9c05456882890f0bb4cd1d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.242048 7328 tablet_service.cc:1467] Processing CreateTablet for tablet 319b287b9c05456882890f0bb4cd1d35 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7a9b82ace20943c99c2d9d15fd3a73c8]), partition=
I20240502 21:26:59.242413 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 319b287b9c05456882890f0bb4cd1d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.246919 7194 tablet_service.cc:1467] Processing CreateTablet for tablet 319b287b9c05456882890f0bb4cd1d35 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7a9b82ace20943c99c2d9d15fd3a73c8]), partition=
I20240502 21:26:59.247359 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 319b287b9c05456882890f0bb4cd1d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.250288 7688 raft_consensus.cc:482] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:59.250447 7688 raft_consensus.cc:504] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:26:59.251010 7732 tablet_bootstrap.cc:492] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:26:59.252635 7732 tablet_bootstrap.cc:654] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.256976 7688 leader_election.cc:288] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:26:59.264469 7732 tablet_bootstrap.cc:492] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:26:59.265606 7732 ts_tablet_manager.cc:1367] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.004s
I20240502 21:26:59.269603 7732 raft_consensus.cc:348] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:26:59.269997 7732 raft_consensus.cc:374] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.270201 7732 raft_consensus.cc:729] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.270591 7732 consensus_queue.cc:260] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:26:59.271261 7732 ts_tablet_manager.cc:1398] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:26:59.279338 7734 tablet_bootstrap.cc:492] T 319b287b9c05456882890f0bb4cd1d35 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:26:59.281587 7734 tablet_bootstrap.cc:654] T 319b287b9c05456882890f0bb4cd1d35 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.282824 7688 raft_consensus.cc:482] T af97ce1401784f45807466e2ebb970f7 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:59.283063 7688 raft_consensus.cc:504] T af97ce1401784f45807466e2ebb970f7 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:26:59.284085 7688 leader_election.cc:288] T af97ce1401784f45807466e2ebb970f7 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:59.284557 7400 raft_consensus.cc:482] T af97ce1401784f45807466e2ebb970f7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:59.284790 7400 raft_consensus.cc:504] T af97ce1401784f45807466e2ebb970f7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:26:59.286139 7400 leader_election.cc:288] T af97ce1401784f45807466e2ebb970f7 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:26:59.287664 7683 tablet_bootstrap.cc:492] T 319b287b9c05456882890f0bb4cd1d35 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:26:59.289765 7683 tablet_bootstrap.cc:654] T 319b287b9c05456882890f0bb4cd1d35 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.328397 7734 tablet_bootstrap.cc:492] T 319b287b9c05456882890f0bb4cd1d35 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:26:59.328841 7734 ts_tablet_manager.cc:1367] T 319b287b9c05456882890f0bb4cd1d35 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.050s user 0.008s sys 0.001s
I20240502 21:26:59.332911 7734 raft_consensus.cc:348] T 319b287b9c05456882890f0bb4cd1d35 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:26:59.337126 7734 raft_consensus.cc:374] T 319b287b9c05456882890f0bb4cd1d35 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.337280 7734 raft_consensus.cc:729] T 319b287b9c05456882890f0bb4cd1d35 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.340004 7734 consensus_queue.cc:260] T 319b287b9c05456882890f0bb4cd1d35 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:26:59.341162 7683 tablet_bootstrap.cc:492] T 319b287b9c05456882890f0bb4cd1d35 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:26:59.343652 7683 ts_tablet_manager.cc:1367] T 319b287b9c05456882890f0bb4cd1d35 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.056s user 0.005s sys 0.007s
W20240502 21:26:59.343830 7396 tablet.cc:2369] T 1d3be8c5eaac423db11b282739cd1d40 P d0456c5e55a542128688904e003d403c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:59.344661 7683 raft_consensus.cc:348] T 319b287b9c05456882890f0bb4cd1d35 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:26:59.344897 7683 raft_consensus.cc:374] T 319b287b9c05456882890f0bb4cd1d35 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.344961 7683 raft_consensus.cc:729] T 319b287b9c05456882890f0bb4cd1d35 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.345207 7683 consensus_queue.cc:260] T 319b287b9c05456882890f0bb4cd1d35 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:26:59.346443 7734 ts_tablet_manager.cc:1398] T 319b287b9c05456882890f0bb4cd1d35 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.017s user 0.000s sys 0.003s
I20240502 21:26:59.362985 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3be8c5eaac423db11b282739cd1d40" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:26:59.363430 7484 raft_consensus.cc:2454] T 1d3be8c5eaac423db11b282739cd1d40 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:26:59.364228 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af97ce1401784f45807466e2ebb970f7" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:26:59.364408 7484 raft_consensus.cc:2454] T af97ce1401784f45807466e2ebb970f7 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:26:59.365011 7148 leader_election.cc:302] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:26:59.365523 7688 raft_consensus.cc:2781] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:59.365620 7688 raft_consensus.cc:482] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:59.365731 7688 raft_consensus.cc:3037] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.368710 7688 raft_consensus.cc:504] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:26:59.369726 7688 leader_election.cc:288] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:26:59.370163 7148 leader_election.cc:302] T af97ce1401784f45807466e2ebb970f7 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:26:59.370731 7688 raft_consensus.cc:2781] T af97ce1401784f45807466e2ebb970f7 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:59.370899 7688 raft_consensus.cc:482] T af97ce1401784f45807466e2ebb970f7 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:59.371045 7688 raft_consensus.cc:3037] T af97ce1401784f45807466e2ebb970f7 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.372254 4295 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240502 21:26:59.372467 4295 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240502 21:26:59.374037 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af97ce1401784f45807466e2ebb970f7" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:26:59.374243 7484 raft_consensus.cc:2454] T af97ce1401784f45807466e2ebb970f7 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:26:59.375658 7688 raft_consensus.cc:504] T af97ce1401784f45807466e2ebb970f7 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:26:59.376338 7688 leader_election.cc:288] T af97ce1401784f45807466e2ebb970f7 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:26:59.376870 4295 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet eaef07076cf1438b8aac5ac1524bfccf
I20240502 21:26:59.377110 4770 leader_election.cc:302] T af97ce1401784f45807466e2ebb970f7 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:59.377243 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3be8c5eaac423db11b282739cd1d40" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:26:59.377398 7400 raft_consensus.cc:2781] T af97ce1401784f45807466e2ebb970f7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:59.377419 7484 raft_consensus.cc:3037] T 1d3be8c5eaac423db11b282739cd1d40 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.377471 4295 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet e0768388c4c542aab05a42d5ebc78f93
I20240502 21:26:59.377521 7400 raft_consensus.cc:482] T af97ce1401784f45807466e2ebb970f7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:59.377625 7400 raft_consensus.cc:3037] T af97ce1401784f45807466e2ebb970f7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.378068 4295 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet f134f962f5024e128db94eea8527712f
I20240502 21:26:59.379632 4553 tablet_service.cc:1514] Processing DeleteTablet for tablet f134f962f5024e128db94eea8527712f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:59 UTC) from {username='slave'} at 127.0.0.1:60018
I20240502 21:26:59.379787 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet eaef07076cf1438b8aac5ac1524bfccf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:59 UTC) from {username='slave'} at 127.0.0.1:60018
I20240502 21:26:59.379956 7717 tablet_replica.cc:330] T f134f962f5024e128db94eea8527712f P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:26:59.380167 7717 raft_consensus.cc:2229] T f134f962f5024e128db94eea8527712f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:59.380410 7717 raft_consensus.cc:2258] T f134f962f5024e128db94eea8527712f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.381146 7717 ts_tablet_manager.cc:1875] T f134f962f5024e128db94eea8527712f P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:59.381019 7400 raft_consensus.cc:504] T af97ce1401784f45807466e2ebb970f7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:26:59.382450 7400 leader_election.cc:288] T af97ce1401784f45807466e2ebb970f7 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:26:59.383033 7683 ts_tablet_manager.cc:1398] T 319b287b9c05456882890f0bb4cd1d35 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.039s user 0.002s sys 0.000s
I20240502 21:26:59.383143 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af97ce1401784f45807466e2ebb970f7" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:26:59.383419 7483 raft_consensus.cc:3037] T af97ce1401784f45807466e2ebb970f7 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.383720 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet e0768388c4c542aab05a42d5ebc78f93 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:59 UTC) from {username='slave'} at 127.0.0.1:44764
I20240502 21:26:59.384016 7719 tablet_replica.cc:330] T e0768388c4c542aab05a42d5ebc78f93 P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:26:59.384248 7719 raft_consensus.cc:2229] T e0768388c4c542aab05a42d5ebc78f93 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:59.384510 7719 raft_consensus.cc:2258] T e0768388c4c542aab05a42d5ebc78f93 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.385087 4423 tablet_service.cc:1514] Processing DeleteTablet for tablet eaef07076cf1438b8aac5ac1524bfccf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:59 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:26:59.385289 7719 ts_tablet_manager.cc:1875] T e0768388c4c542aab05a42d5ebc78f93 P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:59.385404 4422 tablet_service.cc:1514] Processing DeleteTablet for tablet e0768388c4c542aab05a42d5ebc78f93 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:59 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:26:59.385870 7716 tablet_replica.cc:330] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:26:59.386085 7716 raft_consensus.cc:2229] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:59.386452 7483 raft_consensus.cc:2454] T af97ce1401784f45807466e2ebb970f7 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:26:59.386667 7716 raft_consensus.cc:2258] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.386958 4770 leader_election.cc:302] T af97ce1401784f45807466e2ebb970f7 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:26:59.387282 7400 raft_consensus.cc:2781] T af97ce1401784f45807466e2ebb970f7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:59.387391 7716 ts_tablet_manager.cc:1875] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:59.387447 7400 raft_consensus.cc:686] T af97ce1401784f45807466e2ebb970f7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:26:59.387521 7484 raft_consensus.cc:2454] T 1d3be8c5eaac423db11b282739cd1d40 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:26:59.387923 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet f134f962f5024e128db94eea8527712f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:59 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:26:59.387969 7400 consensus_queue.cc:237] T af97ce1401784f45807466e2ebb970f7 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:26:59.388191 4816 tablet_service.cc:1514] Processing DeleteTablet for tablet eaef07076cf1438b8aac5ac1524bfccf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:59 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:26:59.388497 7718 tablet_replica.cc:330] T eaef07076cf1438b8aac5ac1524bfccf P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:26:59.388711 7718 raft_consensus.cc:2229] T eaef07076cf1438b8aac5ac1524bfccf P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:59.388938 7718 raft_consensus.cc:2258] T eaef07076cf1438b8aac5ac1524bfccf P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.389675 7718 ts_tablet_manager.cc:1875] T eaef07076cf1438b8aac5ac1524bfccf P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:59.390017 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af97ce1401784f45807466e2ebb970f7" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:26:59.390378 7484 raft_consensus.cc:2379] T af97ce1401784f45807466e2ebb970f7 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 39981a8dfe94462b9a73ba489af4c714 in current term 1: Already voted for candidate 73a1f2ca1f544de9b0e3b37015f85328 in this term.
I20240502 21:26:59.391733 4818 tablet_service.cc:1514] Processing DeleteTablet for tablet e0768388c4c542aab05a42d5ebc78f93 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:59 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:26:59.391983 4817 tablet_service.cc:1514] Processing DeleteTablet for tablet f134f962f5024e128db94eea8527712f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:59 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:26:59.395152 7148 leader_election.cc:302] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:26:59.395958 7688 raft_consensus.cc:2781] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:59.399205 7717 ts_tablet_manager.cc:1888] T f134f962f5024e128db94eea8527712f P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:59.401929 7716 ts_tablet_manager.cc:1888] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:59.402168 7717 log.cc:1196] T f134f962f5024e128db94eea8527712f P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/f134f962f5024e128db94eea8527712f
I20240502 21:26:59.402279 7718 ts_tablet_manager.cc:1888] T eaef07076cf1438b8aac5ac1524bfccf P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:59.402434 7718 log.cc:1196] T eaef07076cf1438b8aac5ac1524bfccf P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/eaef07076cf1438b8aac5ac1524bfccf
I20240502 21:26:59.402889 7717 ts_tablet_manager.cc:1909] T f134f962f5024e128db94eea8527712f P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting consensus metadata
I20240502 21:26:59.402905 7718 ts_tablet_manager.cc:1909] T eaef07076cf1438b8aac5ac1524bfccf P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:26:59.403075 7716 log.cc:1196] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/eaef07076cf1438b8aac5ac1524bfccf
I20240502 21:26:59.403491 7718 tablet_replica.cc:330] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:26:59.403620 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet eaef07076cf1438b8aac5ac1524bfccf (table rebalance_test_table_2 [id=d26a56879c874f3cb4fbd05df61cd275]) successfully deleted
I20240502 21:26:59.404211 7719 ts_tablet_manager.cc:1888] T e0768388c4c542aab05a42d5ebc78f93 P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:59.404361 7719 log.cc:1196] T e0768388c4c542aab05a42d5ebc78f93 P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/e0768388c4c542aab05a42d5ebc78f93
I20240502 21:26:59.404834 7719 ts_tablet_manager.cc:1909] T e0768388c4c542aab05a42d5ebc78f93 P 398c3c8f3195492c900439ad5e1931bb: Deleting consensus metadata
I20240502 21:26:59.404944 7716 ts_tablet_manager.cc:1909] T eaef07076cf1438b8aac5ac1524bfccf P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:26:59.405086 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3be8c5eaac423db11b282739cd1d40" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:26:59.405715 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet f134f962f5024e128db94eea8527712f (table rebalance_test_table_2 [id=d26a56879c874f3cb4fbd05df61cd275]) successfully deleted
I20240502 21:26:59.405809 7350 raft_consensus.cc:2454] T 1d3be8c5eaac423db11b282739cd1d40 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:26:59.405978 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet e0768388c4c542aab05a42d5ebc78f93 (table rebalance_test_table_2 [id=d26a56879c874f3cb4fbd05df61cd275]) successfully deleted
I20240502 21:26:59.406795 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3be8c5eaac423db11b282739cd1d40" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:26:59.407059 7350 raft_consensus.cc:3037] T 1d3be8c5eaac423db11b282739cd1d40 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.406926 7717 tablet_replica.cc:330] T eaef07076cf1438b8aac5ac1524bfccf P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:26:59.407503 7717 raft_consensus.cc:2229] T eaef07076cf1438b8aac5ac1524bfccf P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:59.408179 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet eaef07076cf1438b8aac5ac1524bfccf (table rebalance_test_table_2 [id=d26a56879c874f3cb4fbd05df61cd275]) successfully deleted
I20240502 21:26:59.408576 7717 raft_consensus.cc:2258] T eaef07076cf1438b8aac5ac1524bfccf P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.408843 7716 tablet_replica.cc:330] T e0768388c4c542aab05a42d5ebc78f93 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:26:59.409130 7716 raft_consensus.cc:2229] T e0768388c4c542aab05a42d5ebc78f93 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:59.409936 7717 ts_tablet_manager.cc:1875] T eaef07076cf1438b8aac5ac1524bfccf P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:59.410216 7716 raft_consensus.cc:2258] T e0768388c4c542aab05a42d5ebc78f93 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.410882 7350 raft_consensus.cc:2454] T 1d3be8c5eaac423db11b282739cd1d40 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:26:59.412216 7716 ts_tablet_manager.cc:1875] T e0768388c4c542aab05a42d5ebc78f93 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:59.413017 7718 raft_consensus.cc:2229] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:59.414392 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af97ce1401784f45807466e2ebb970f7" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:26:59.414819 7215 raft_consensus.cc:2379] T af97ce1401784f45807466e2ebb970f7 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 73a1f2ca1f544de9b0e3b37015f85328 in current term 1: Already voted for candidate 39981a8dfe94462b9a73ba489af4c714 in this term.
I20240502 21:26:59.414597 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af97ce1401784f45807466e2ebb970f7" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:26:59.418625 7688 raft_consensus.cc:686] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:26:59.419212 7688 consensus_queue.cc:237] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:26:59.417644 7718 raft_consensus.cc:2258] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.421849 7718 ts_tablet_manager.cc:1875] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:59.421897 4295 catalog_manager.cc:5484] T af97ce1401784f45807466e2ebb970f7 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:26:59.433303 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af97ce1401784f45807466e2ebb970f7" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:26:59.433897 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af97ce1401784f45807466e2ebb970f7" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:26:59.434737 7148 leader_election.cc:302] T af97ce1401784f45807466e2ebb970f7 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714; no voters: 31a72286397c4edc9a63eca3412be168, 73a1f2ca1f544de9b0e3b37015f85328
I20240502 21:26:59.435065 7783 raft_consensus.cc:2726] T af97ce1401784f45807466e2ebb970f7 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 21:26:59.437139 4295 catalog_manager.cc:5484] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69). New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:26:59.437754 7718 ts_tablet_manager.cc:1888] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:59.438066 7718 log.cc:1196] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/e0768388c4c542aab05a42d5ebc78f93
I20240502 21:26:59.438735 7718 ts_tablet_manager.cc:1909] T e0768388c4c542aab05a42d5ebc78f93 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:26:59.439818 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet e0768388c4c542aab05a42d5ebc78f93 (table rebalance_test_table_2 [id=d26a56879c874f3cb4fbd05df61cd275]) successfully deleted
I20240502 21:26:59.443447 7717 ts_tablet_manager.cc:1888] T eaef07076cf1438b8aac5ac1524bfccf P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:59.444561 7718 tablet_replica.cc:330] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:26:59.444767 7718 raft_consensus.cc:2229] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:59.445140 7718 raft_consensus.cc:2258] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.447882 7718 ts_tablet_manager.cc:1875] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:59.454568 7717 log.cc:1196] T eaef07076cf1438b8aac5ac1524bfccf P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/eaef07076cf1438b8aac5ac1524bfccf
I20240502 21:26:59.455102 7717 ts_tablet_manager.cc:1909] T eaef07076cf1438b8aac5ac1524bfccf P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting consensus metadata
I20240502 21:26:59.456321 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet eaef07076cf1438b8aac5ac1524bfccf (table rebalance_test_table_2 [id=d26a56879c874f3cb4fbd05df61cd275]) successfully deleted
I20240502 21:26:59.459522 7716 ts_tablet_manager.cc:1888] T e0768388c4c542aab05a42d5ebc78f93 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:59.460249 7716 log.cc:1196] T e0768388c4c542aab05a42d5ebc78f93 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/e0768388c4c542aab05a42d5ebc78f93
I20240502 21:26:59.460733 7716 ts_tablet_manager.cc:1909] T e0768388c4c542aab05a42d5ebc78f93 P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:26:59.461859 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet e0768388c4c542aab05a42d5ebc78f93 (table rebalance_test_table_2 [id=d26a56879c874f3cb4fbd05df61cd275]) successfully deleted
I20240502 21:26:59.468839 7716 tablet_replica.cc:330] T f134f962f5024e128db94eea8527712f P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:26:59.469115 7716 raft_consensus.cc:2229] T f134f962f5024e128db94eea8527712f P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:59.469378 7716 raft_consensus.cc:2258] T f134f962f5024e128db94eea8527712f P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.470197 7716 ts_tablet_manager.cc:1875] T f134f962f5024e128db94eea8527712f P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:59.474786 7718 ts_tablet_manager.cc:1888] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:59.474943 7718 log.cc:1196] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/f134f962f5024e128db94eea8527712f
I20240502 21:26:59.475445 7718 ts_tablet_manager.cc:1909] T f134f962f5024e128db94eea8527712f P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:26:59.476704 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet f134f962f5024e128db94eea8527712f (table rebalance_test_table_2 [id=d26a56879c874f3cb4fbd05df61cd275]) successfully deleted
I20240502 21:26:59.483464 7716 ts_tablet_manager.cc:1888] T f134f962f5024e128db94eea8527712f P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:59.483646 7716 log.cc:1196] T f134f962f5024e128db94eea8527712f P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/f134f962f5024e128db94eea8527712f
I20240502 21:26:59.484098 7716 ts_tablet_manager.cc:1909] T f134f962f5024e128db94eea8527712f P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:26:59.485404 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet f134f962f5024e128db94eea8527712f (table rebalance_test_table_2 [id=d26a56879c874f3cb4fbd05df61cd275]) successfully deleted
I20240502 21:26:59.492203 7735 raft_consensus.cc:482] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:59.492442 7735 raft_consensus.cc:504] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:26:59.493340 7735 leader_election.cc:288] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:26:59.494479 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "319b287b9c05456882890f0bb4cd1d35" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:26:59.494771 7214 raft_consensus.cc:2454] T 319b287b9c05456882890f0bb4cd1d35 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:26:59.495682 7551 leader_election.cc:302] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:26:59.498450 7735 raft_consensus.cc:2781] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:59.498572 7735 raft_consensus.cc:482] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:59.498705 7735 raft_consensus.cc:3037] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.501602 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "319b287b9c05456882890f0bb4cd1d35" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:26:59.501895 7350 raft_consensus.cc:2454] T 319b287b9c05456882890f0bb4cd1d35 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:26:59.503624 7735 raft_consensus.cc:504] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:26:59.504674 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "319b287b9c05456882890f0bb4cd1d35" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:26:59.504822 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "319b287b9c05456882890f0bb4cd1d35" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:26:59.504909 7350 raft_consensus.cc:3037] T 319b287b9c05456882890f0bb4cd1d35 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.504369 7735 leader_election.cc:288] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:26:59.505067 7214 raft_consensus.cc:3037] T 319b287b9c05456882890f0bb4cd1d35 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.507656 7214 raft_consensus.cc:2454] T 319b287b9c05456882890f0bb4cd1d35 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:26:59.508720 7551 leader_election.cc:302] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:26:59.509063 7735 raft_consensus.cc:2781] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:59.509289 7735 raft_consensus.cc:686] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Becoming Leader. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Running, Role: LEADER
I20240502 21:26:59.509724 7735 consensus_queue.cc:237] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:26:59.531967 4291 catalog_manager.cc:5484] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ddd486109f5434cace13d60d633ac04 (127.29.82.72). New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:26:59.533826 7597 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:59.507710 7350 raft_consensus.cc:2454] T 319b287b9c05456882890f0bb4cd1d35 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:26:59.546154 7194 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:59.547361 7464 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:59.557070 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:26:59.601449 4421 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:26:59.616036 4817 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:26:59.626698 4555 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:26:59.662478 7328 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:59.674852 4291 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:26:59.684546 7350 raft_consensus.cc:1261] T 319b287b9c05456882890f0bb4cd1d35 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:59.685314 7214 raft_consensus.cc:1261] T 319b287b9c05456882890f0bb4cd1d35 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:59.686295 7755 consensus_queue.cc:1035] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:59.687839 7735 consensus_queue.cc:1035] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:59.687834 7755 consensus_queue.cc:1035] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:26:59.696846 7735 consensus_queue.cc:1035] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:59.724720 4300 catalog_manager.cc:5576] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719) does not have the latest schema for tablet 319b287b9c05456882890f0bb4cd1d35 (table rebalancer_dynamic_table [id=7a9b82ace20943c99c2d9d15fd3a73c8]). Expected version 1 got 0
I20240502 21:26:59.726395 4300 catalog_manager.cc:5576] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193) does not have the latest schema for tablet 319b287b9c05456882890f0bb4cd1d35 (table rebalancer_dynamic_table [id=7a9b82ace20943c99c2d9d15fd3a73c8]). Expected version 1 got 0
I20240502 21:26:59.741111 7803 mvcc.cc:205] Tried to move back new op lower bound from 7023350659813740544 to 7023350659114733568. Current Snapshot: MvccSnapshot[applied={T|T < 7023350659813740544}]
I20240502 21:26:59.741451 7813 tablet.cc:1722] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:59.743207 7813 tablet.cc:1620] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04: MemRowSet was empty: no flush needed.
I20240502 21:26:59.745692 7806 mvcc.cc:205] Tried to move back new op lower bound from 7023350659813740544 to 7023350659114733568. Current Snapshot: MvccSnapshot[applied={T|T < 7023350659813740544}]
I20240502 21:26:59.745270 7804 mvcc.cc:205] Tried to move back new op lower bound from 7023350659813740544 to 7023350659114733568. Current Snapshot: MvccSnapshot[applied={T|T < 7023350659813740544}]
I20240502 21:26:59.756307 4300 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=7a9b82ace20943c99c2d9d15fd3a73c8] alter complete (version 1)
I20240502 21:26:59.757246 7814 tablet.cc:1722] T 319b287b9c05456882890f0bb4cd1d35 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:59.757864 7814 tablet.cc:1620] T 319b287b9c05456882890f0bb4cd1d35 P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:26:59.771492 7815 tablet.cc:1722] T 319b287b9c05456882890f0bb4cd1d35 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:59.776074 4300 catalog_manager.cc:5576] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193) does not have the latest schema for tablet 319b287b9c05456882890f0bb4cd1d35 (table rebalancer_dynamic_table [id=7a9b82ace20943c99c2d9d15fd3a73c8]). Expected version 1 got 0
I20240502 21:26:59.777503 4291 catalog_manager.cc:5576] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719) does not have the latest schema for tablet 319b287b9c05456882890f0bb4cd1d35 (table rebalancer_dynamic_table [id=7a9b82ace20943c99c2d9d15fd3a73c8]). Expected version 1 got 0
I20240502 21:26:59.782667 7815 tablet.cc:1620] T 319b287b9c05456882890f0bb4cd1d35 P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:26:59.854990 7822 consensus_queue.cc:1035] T af97ce1401784f45807466e2ebb970f7 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:59.860983 7822 consensus_queue.cc:1035] T af97ce1401784f45807466e2ebb970f7 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:59.867369 4300 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:26:59.876168 7813 tablet.cc:1722] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:59.876678 7813 tablet.cc:1620] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04: MemRowSet was empty: no flush needed.
I20240502 21:26:59.877169 7814 tablet.cc:1722] T 319b287b9c05456882890f0bb4cd1d35 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:59.877671 7814 tablet.cc:1620] T 319b287b9c05456882890f0bb4cd1d35 P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:26:59.877636 7815 tablet.cc:1722] T 319b287b9c05456882890f0bb4cd1d35 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:59.878113 7815 tablet.cc:1620] T 319b287b9c05456882890f0bb4cd1d35 P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:26:59.886469 4291 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=7a9b82ace20943c99c2d9d15fd3a73c8] alter complete (version 2)
I20240502 21:26:59.893795 4300 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240502 21:26:59.894016 4300 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240502 21:26:59.897990 4300 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 20c21d242c8c4e00a45ae5de7388db3d
I20240502 21:26:59.898543 4300 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 18039778dfde4bcaa1245544e3d3c1a3
I20240502 21:26:59.898845 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet 20c21d242c8c4e00a45ae5de7388db3d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:59 UTC) from {username='slave'} at 127.0.0.1:60018
I20240502 21:26:59.899049 4300 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet f06f4eacdcec4b86a1bfd8e79be58bfe
I20240502 21:26:59.899039 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet 20c21d242c8c4e00a45ae5de7388db3d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:59 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:26:59.899168 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet 20c21d242c8c4e00a45ae5de7388db3d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:59 UTC) from {username='slave'} at 127.0.0.1:44764
I20240502 21:26:59.899314 7717 tablet_replica.cc:330] T 20c21d242c8c4e00a45ae5de7388db3d P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:26:59.899434 7719 tablet_replica.cc:330] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:26:59.899461 7716 tablet_replica.cc:330] T 20c21d242c8c4e00a45ae5de7388db3d P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:26:59.899541 7717 raft_consensus.cc:2229] T 20c21d242c8c4e00a45ae5de7388db3d P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:59.899701 7719 raft_consensus.cc:2229] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:59.899715 7716 raft_consensus.cc:2229] T 20c21d242c8c4e00a45ae5de7388db3d P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:59.899823 7717 raft_consensus.cc:2258] T 20c21d242c8c4e00a45ae5de7388db3d P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.899963 7716 raft_consensus.cc:2258] T 20c21d242c8c4e00a45ae5de7388db3d P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.900044 7719 raft_consensus.cc:2258] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.900381 7717 ts_tablet_manager.cc:1875] T 20c21d242c8c4e00a45ae5de7388db3d P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:59.900724 7716 ts_tablet_manager.cc:1875] T 20c21d242c8c4e00a45ae5de7388db3d P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:59.900763 7719 ts_tablet_manager.cc:1875] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:59.903962 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet 18039778dfde4bcaa1245544e3d3c1a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:59 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:26:59.904174 4817 tablet_service.cc:1514] Processing DeleteTablet for tablet 18039778dfde4bcaa1245544e3d3c1a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:59 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:26:59.904585 7718 tablet_replica.cc:330] T 18039778dfde4bcaa1245544e3d3c1a3 P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:26:59.904831 7718 raft_consensus.cc:2229] T 18039778dfde4bcaa1245544e3d3c1a3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:59.905092 7718 raft_consensus.cc:2258] T 18039778dfde4bcaa1245544e3d3c1a3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.905711 7718 ts_tablet_manager.cc:1875] T 18039778dfde4bcaa1245544e3d3c1a3 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:59.907838 7716 ts_tablet_manager.cc:1888] T 20c21d242c8c4e00a45ae5de7388db3d P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:59.907982 7716 log.cc:1196] T 20c21d242c8c4e00a45ae5de7388db3d P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/20c21d242c8c4e00a45ae5de7388db3d
I20240502 21:26:59.908378 7716 ts_tablet_manager.cc:1909] T 20c21d242c8c4e00a45ae5de7388db3d P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:26:59.908864 7719 ts_tablet_manager.cc:1888] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:59.909025 7719 log.cc:1196] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/20c21d242c8c4e00a45ae5de7388db3d
I20240502 21:26:59.909346 7716 tablet_replica.cc:330] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:26:59.909471 7719 ts_tablet_manager.cc:1909] T 20c21d242c8c4e00a45ae5de7388db3d P 398c3c8f3195492c900439ad5e1931bb: Deleting consensus metadata
I20240502 21:26:59.909601 7716 raft_consensus.cc:2229] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:59.909958 7716 raft_consensus.cc:2258] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.910493 7718 ts_tablet_manager.cc:1888] T 18039778dfde4bcaa1245544e3d3c1a3 P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:59.910637 7718 log.cc:1196] T 18039778dfde4bcaa1245544e3d3c1a3 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/18039778dfde4bcaa1245544e3d3c1a3
I20240502 21:26:59.910838 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet 18039778dfde4bcaa1245544e3d3c1a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:59 UTC) from {username='slave'} at 127.0.0.1:60018
I20240502 21:26:59.910885 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet 20c21d242c8c4e00a45ae5de7388db3d (table rebalance_test_table_4 [id=2f224bc51e0b4008a30114b6f180f43b]) successfully deleted
I20240502 21:26:59.911092 7718 ts_tablet_manager.cc:1909] T 18039778dfde4bcaa1245544e3d3c1a3 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:26:59.911180 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet f06f4eacdcec4b86a1bfd8e79be58bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:59 UTC) from {username='slave'} at 127.0.0.1:44764
I20240502 21:26:59.911468 7719 tablet_replica.cc:330] T f06f4eacdcec4b86a1bfd8e79be58bfe P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:26:59.911459 7716 ts_tablet_manager.cc:1875] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:59.911687 7719 raft_consensus.cc:2229] T f06f4eacdcec4b86a1bfd8e79be58bfe P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:59.911751 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet 20c21d242c8c4e00a45ae5de7388db3d (table rebalance_test_table_4 [id=2f224bc51e0b4008a30114b6f180f43b]) successfully deleted
I20240502 21:26:59.912179 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet f06f4eacdcec4b86a1bfd8e79be58bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:59 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:26:59.912442 7719 raft_consensus.cc:2258] T f06f4eacdcec4b86a1bfd8e79be58bfe P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.912757 4817 tablet_service.cc:1514] Processing DeleteTablet for tablet f06f4eacdcec4b86a1bfd8e79be58bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:59 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:26:59.914525 7719 ts_tablet_manager.cc:1875] T f06f4eacdcec4b86a1bfd8e79be58bfe P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:59.914685 7717 ts_tablet_manager.cc:1888] T 20c21d242c8c4e00a45ae5de7388db3d P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:59.914837 7717 log.cc:1196] T 20c21d242c8c4e00a45ae5de7388db3d P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/20c21d242c8c4e00a45ae5de7388db3d
I20240502 21:26:59.915212 7717 ts_tablet_manager.cc:1909] T 20c21d242c8c4e00a45ae5de7388db3d P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting consensus metadata
I20240502 21:26:59.916066 7717 tablet_replica.cc:330] T 18039778dfde4bcaa1245544e3d3c1a3 P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:26:59.916141 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet 20c21d242c8c4e00a45ae5de7388db3d (table rebalance_test_table_4 [id=2f224bc51e0b4008a30114b6f180f43b]) successfully deleted
I20240502 21:26:59.916292 7717 raft_consensus.cc:2229] T 18039778dfde4bcaa1245544e3d3c1a3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:59.916510 7717 raft_consensus.cc:2258] T 18039778dfde4bcaa1245544e3d3c1a3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.917094 7717 ts_tablet_manager.cc:1875] T 18039778dfde4bcaa1245544e3d3c1a3 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:59.917634 7716 ts_tablet_manager.cc:1888] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:59.917783 7716 log.cc:1196] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/18039778dfde4bcaa1245544e3d3c1a3
I20240502 21:26:59.918243 7716 ts_tablet_manager.cc:1909] T 18039778dfde4bcaa1245544e3d3c1a3 P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:26:59.919157 7716 tablet_replica.cc:330] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:26:59.919333 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet 18039778dfde4bcaa1245544e3d3c1a3 (table rebalance_test_table_4 [id=2f224bc51e0b4008a30114b6f180f43b]) successfully deleted
I20240502 21:26:59.919384 7716 raft_consensus.cc:2229] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:59.919770 7716 raft_consensus.cc:2258] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.920181 7718 tablet_replica.cc:330] T f06f4eacdcec4b86a1bfd8e79be58bfe P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:26:59.920264 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet 18039778dfde4bcaa1245544e3d3c1a3 (table rebalance_test_table_4 [id=2f224bc51e0b4008a30114b6f180f43b]) successfully deleted
I20240502 21:26:59.920480 7716 ts_tablet_manager.cc:1875] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:59.920506 7718 raft_consensus.cc:2229] T f06f4eacdcec4b86a1bfd8e79be58bfe P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:59.920850 7718 raft_consensus.cc:2258] T f06f4eacdcec4b86a1bfd8e79be58bfe P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.921592 7718 ts_tablet_manager.cc:1875] T f06f4eacdcec4b86a1bfd8e79be58bfe P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:59.924422 7719 ts_tablet_manager.cc:1888] T f06f4eacdcec4b86a1bfd8e79be58bfe P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:59.924599 7719 log.cc:1196] T f06f4eacdcec4b86a1bfd8e79be58bfe P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/f06f4eacdcec4b86a1bfd8e79be58bfe
I20240502 21:26:59.925176 7719 ts_tablet_manager.cc:1909] T f06f4eacdcec4b86a1bfd8e79be58bfe P 398c3c8f3195492c900439ad5e1931bb: Deleting consensus metadata
I20240502 21:26:59.926185 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet f06f4eacdcec4b86a1bfd8e79be58bfe (table rebalance_test_table_4 [id=2f224bc51e0b4008a30114b6f180f43b]) successfully deleted
I20240502 21:26:59.926308 7716 ts_tablet_manager.cc:1888] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:59.926434 7716 log.cc:1196] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/f06f4eacdcec4b86a1bfd8e79be58bfe
I20240502 21:26:59.926844 7716 ts_tablet_manager.cc:1909] T f06f4eacdcec4b86a1bfd8e79be58bfe P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:26:59.927954 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet f06f4eacdcec4b86a1bfd8e79be58bfe (table rebalance_test_table_4 [id=2f224bc51e0b4008a30114b6f180f43b]) successfully deleted
I20240502 21:26:59.928246 7718 ts_tablet_manager.cc:1888] T f06f4eacdcec4b86a1bfd8e79be58bfe P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:59.928411 7718 log.cc:1196] T f06f4eacdcec4b86a1bfd8e79be58bfe P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/f06f4eacdcec4b86a1bfd8e79be58bfe
I20240502 21:26:59.928875 7718 ts_tablet_manager.cc:1909] T f06f4eacdcec4b86a1bfd8e79be58bfe P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:26:59.929934 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet f06f4eacdcec4b86a1bfd8e79be58bfe (table rebalance_test_table_4 [id=2f224bc51e0b4008a30114b6f180f43b]) successfully deleted
I20240502 21:26:59.932659 7717 ts_tablet_manager.cc:1888] T 18039778dfde4bcaa1245544e3d3c1a3 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:59.932792 7717 log.cc:1196] T 18039778dfde4bcaa1245544e3d3c1a3 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/18039778dfde4bcaa1245544e3d3c1a3
I20240502 21:26:59.933108 7717 ts_tablet_manager.cc:1909] T 18039778dfde4bcaa1245544e3d3c1a3 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting consensus metadata
I20240502 21:26:59.933928 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet 18039778dfde4bcaa1245544e3d3c1a3 (table rebalance_test_table_4 [id=2f224bc51e0b4008a30114b6f180f43b]) successfully deleted
I20240502 21:26:59.974220 4300 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:59.986372 7464 tablet_service.cc:1467] Processing CreateTablet for tablet ba5b4c81211647dd8383ec9484bb9c75 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=1b458a4121624eaaaf47feadd84aab8c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:59.986775 7597 tablet_service.cc:1467] Processing CreateTablet for tablet ba5b4c81211647dd8383ec9484bb9c75 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=1b458a4121624eaaaf47feadd84aab8c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:59.987051 7464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba5b4c81211647dd8383ec9484bb9c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.987316 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba5b4c81211647dd8383ec9484bb9c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.988529 7596 tablet_service.cc:1467] Processing CreateTablet for tablet f800bfd495a94f2a952bc57a348304d0 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=1b458a4121624eaaaf47feadd84aab8c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:59.988994 7596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f800bfd495a94f2a952bc57a348304d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.989010 7463 tablet_service.cc:1467] Processing CreateTablet for tablet f800bfd495a94f2a952bc57a348304d0 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=1b458a4121624eaaaf47feadd84aab8c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:59.989544 7463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f800bfd495a94f2a952bc57a348304d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.992446 4686 tablet_service.cc:1467] Processing CreateTablet for tablet ba5b4c81211647dd8383ec9484bb9c75 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=1b458a4121624eaaaf47feadd84aab8c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:59.992935 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba5b4c81211647dd8383ec9484bb9c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.995653 4292 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:26:59.999346 7857 tablet_bootstrap.cc:492] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:00.000376 7194 tablet_service.cc:1467] Processing CreateTablet for tablet f800bfd495a94f2a952bc57a348304d0 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=1b458a4121624eaaaf47feadd84aab8c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.000844 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f800bfd495a94f2a952bc57a348304d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.004053 7857 tablet_bootstrap.cc:654] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.005343 7595 tablet_service.cc:1467] Processing CreateTablet for tablet d095a5e8f1f34b5ea2b6797ef06f79c6 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=1b458a4121624eaaaf47feadd84aab8c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.005679 7858 tablet_bootstrap.cc:492] T f800bfd495a94f2a952bc57a348304d0 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:00.005781 7595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d095a5e8f1f34b5ea2b6797ef06f79c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.008831 7858 tablet_bootstrap.cc:654] T f800bfd495a94f2a952bc57a348304d0 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.015421 7813 tablet.cc:1722] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:00.016862 7858 tablet_bootstrap.cc:492] T f800bfd495a94f2a952bc57a348304d0 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:00.017580 7858 ts_tablet_manager.cc:1367] T f800bfd495a94f2a952bc57a348304d0 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20240502 21:27:00.019912 7328 tablet_service.cc:1467] Processing CreateTablet for tablet d095a5e8f1f34b5ea2b6797ef06f79c6 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=1b458a4121624eaaaf47feadd84aab8c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.020429 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d095a5e8f1f34b5ea2b6797ef06f79c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.020002 7858 raft_consensus.cc:348] T f800bfd495a94f2a952bc57a348304d0 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:00.020763 7858 raft_consensus.cc:374] T f800bfd495a94f2a952bc57a348304d0 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.020859 7858 raft_consensus.cc:729] T f800bfd495a94f2a952bc57a348304d0 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.021538 7858 consensus_queue.cc:260] T f800bfd495a94f2a952bc57a348304d0 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:00.023131 7858 ts_tablet_manager.cc:1398] T f800bfd495a94f2a952bc57a348304d0 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 21:27:00.024073 7688 consensus_queue.cc:1035] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.025353 7195 tablet_service.cc:1467] Processing CreateTablet for tablet d095a5e8f1f34b5ea2b6797ef06f79c6 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=1b458a4121624eaaaf47feadd84aab8c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.025920 7195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d095a5e8f1f34b5ea2b6797ef06f79c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.027911 7814 tablet.cc:1722] T 319b287b9c05456882890f0bb4cd1d35 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:00.028822 7815 tablet.cc:1722] T 319b287b9c05456882890f0bb4cd1d35 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:00.038314 7688 consensus_queue.cc:1035] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.052650 4292 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=7a9b82ace20943c99c2d9d15fd3a73c8] alter complete (version 3)
I20240502 21:27:00.056948 7858 tablet_bootstrap.cc:492] T ba5b4c81211647dd8383ec9484bb9c75 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:00.058995 7858 tablet_bootstrap.cc:654] T ba5b4c81211647dd8383ec9484bb9c75 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.070199 7866 tablet_bootstrap.cc:492] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:00.070894 7863 tablet_bootstrap.cc:492] T f800bfd495a94f2a952bc57a348304d0 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:00.073606 7863 tablet_bootstrap.cc:654] T f800bfd495a94f2a952bc57a348304d0 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.075197 7865 tablet_bootstrap.cc:492] T ba5b4c81211647dd8383ec9484bb9c75 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:00.075805 7866 tablet_bootstrap.cc:654] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.076987 7865 tablet_bootstrap.cc:654] T ba5b4c81211647dd8383ec9484bb9c75 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.084827 7865 tablet_bootstrap.cc:492] T ba5b4c81211647dd8383ec9484bb9c75 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:27:00.085053 7865 ts_tablet_manager.cc:1367] T ba5b4c81211647dd8383ec9484bb9c75 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240502 21:27:00.085978 7865 raft_consensus.cc:348] T ba5b4c81211647dd8383ec9484bb9c75 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:00.086580 7865 raft_consensus.cc:374] T ba5b4c81211647dd8383ec9484bb9c75 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.086702 7865 raft_consensus.cc:729] T ba5b4c81211647dd8383ec9484bb9c75 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.086997 7865 consensus_queue.cc:260] T ba5b4c81211647dd8383ec9484bb9c75 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:00.087709 7865 ts_tablet_manager.cc:1398] T ba5b4c81211647dd8383ec9484bb9c75 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:00.087862 7863 tablet_bootstrap.cc:492] T f800bfd495a94f2a952bc57a348304d0 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:00.088084 7863 ts_tablet_manager.cc:1367] T f800bfd495a94f2a952bc57a348304d0 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240502 21:27:00.089248 7863 raft_consensus.cc:348] T f800bfd495a94f2a952bc57a348304d0 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:00.089663 7863 raft_consensus.cc:374] T f800bfd495a94f2a952bc57a348304d0 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.089673 7858 tablet_bootstrap.cc:492] T ba5b4c81211647dd8383ec9484bb9c75 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:00.089855 7863 raft_consensus.cc:729] T f800bfd495a94f2a952bc57a348304d0 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.089900 7858 ts_tablet_manager.cc:1367] T ba5b4c81211647dd8383ec9484bb9c75 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.002s
I20240502 21:27:00.090211 7863 consensus_queue.cc:260] T f800bfd495a94f2a952bc57a348304d0 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:00.090638 7858 raft_consensus.cc:348] T ba5b4c81211647dd8383ec9484bb9c75 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:00.090970 7858 raft_consensus.cc:374] T ba5b4c81211647dd8383ec9484bb9c75 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.091078 7858 raft_consensus.cc:729] T ba5b4c81211647dd8383ec9484bb9c75 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.091378 7858 consensus_queue.cc:260] T ba5b4c81211647dd8383ec9484bb9c75 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:00.091985 7863 ts_tablet_manager.cc:1398] T f800bfd495a94f2a952bc57a348304d0 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:00.092674 7866 tablet_bootstrap.cc:492] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:00.093130 7863 tablet_bootstrap.cc:492] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:00.093566 7866 ts_tablet_manager.cc:1367] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.003s
I20240502 21:27:00.097682 7863 tablet_bootstrap.cc:654] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.098060 7866 raft_consensus.cc:348] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:00.099144 7866 raft_consensus.cc:374] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.099486 7866 raft_consensus.cc:729] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.100256 7866 consensus_queue.cc:260] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:00.100600 7858 ts_tablet_manager.cc:1398] T ba5b4c81211647dd8383ec9484bb9c75 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 21:27:00.103961 7858 tablet_bootstrap.cc:492] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:00.105475 7866 ts_tablet_manager.cc:1398] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240502 21:27:00.105894 7858 tablet_bootstrap.cc:654] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.108402 7857 tablet_bootstrap.cc:492] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:00.108636 7857 ts_tablet_manager.cc:1367] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.109s user 0.003s sys 0.025s
I20240502 21:27:00.109544 7857 raft_consensus.cc:348] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:00.109827 7857 raft_consensus.cc:374] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.109961 7857 raft_consensus.cc:729] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.110312 7857 consensus_queue.cc:260] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:00.113878 7857 ts_tablet_manager.cc:1398] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240502 21:27:00.114240 7857 tablet_bootstrap.cc:492] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:00.116954 7857 tablet_bootstrap.cc:654] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.119007 7863 tablet_bootstrap.cc:492] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:00.120210 7863 ts_tablet_manager.cc:1367] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.005s
I20240502 21:27:00.121258 7863 raft_consensus.cc:348] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:00.122237 7863 raft_consensus.cc:374] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.122375 7863 raft_consensus.cc:729] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.122821 7858 tablet_bootstrap.cc:492] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:00.122727 7863 consensus_queue.cc:260] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:00.123070 7858 ts_tablet_manager.cc:1367] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20240502 21:27:00.123348 7863 ts_tablet_manager.cc:1398] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:00.124001 7858 raft_consensus.cc:348] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:00.124305 7858 raft_consensus.cc:374] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.124459 7858 raft_consensus.cc:729] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.124795 7858 consensus_queue.cc:260] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:00.126215 7858 ts_tablet_manager.cc:1398] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:00.127344 7857 tablet_bootstrap.cc:492] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:00.127521 7857 ts_tablet_manager.cc:1367] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.005s
I20240502 21:27:00.128899 7857 raft_consensus.cc:348] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:00.129154 7857 raft_consensus.cc:374] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.129237 7857 raft_consensus.cc:729] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.129531 7857 consensus_queue.cc:260] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:00.130034 7857 ts_tablet_manager.cc:1398] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:00.137498 7832 raft_consensus.cc:482] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:00.137637 7832 raft_consensus.cc:504] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:00.138566 7832 leader_election.cc:288] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:27:00.151170 4300 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:00.151436 4300 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:00.152236 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba5b4c81211647dd8383ec9484bb9c75" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:00.152768 7618 raft_consensus.cc:2454] T ba5b4c81211647dd8383ec9484bb9c75 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:00.153645 7418 leader_election.cc:302] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168; no voters:
I20240502 21:27:00.154114 7832 raft_consensus.cc:2781] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:00.154263 7832 raft_consensus.cc:482] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:00.154361 7832 raft_consensus.cc:3037] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.156590 4300 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 319b287b9c05456882890f0bb4cd1d35
I20240502 21:27:00.157550 7595 tablet_service.cc:1514] Processing DeleteTablet for tablet 319b287b9c05456882890f0bb4cd1d35 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:00 UTC) from {username='slave'} at 127.0.0.1:44444
I20240502 21:27:00.158344 7832 raft_consensus.cc:504] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:00.159485 7832 leader_election.cc:288] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Requested vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:27:00.160262 7194 tablet_service.cc:1514] Processing DeleteTablet for tablet 319b287b9c05456882890f0bb4cd1d35 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:00 UTC) from {username='slave'} at 127.0.0.1:44652
I20240502 21:27:00.160614 7328 tablet_service.cc:1514] Processing DeleteTablet for tablet 319b287b9c05456882890f0bb4cd1d35 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:00 UTC) from {username='slave'} at 127.0.0.1:47078
I20240502 21:27:00.160769 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba5b4c81211647dd8383ec9484bb9c75" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:00.160964 7618 raft_consensus.cc:3037] T ba5b4c81211647dd8383ec9484bb9c75 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.166671 7618 raft_consensus.cc:2454] T ba5b4c81211647dd8383ec9484bb9c75 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:00.167909 7738 raft_consensus.cc:482] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:00.168140 7738 raft_consensus.cc:504] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:00.170434 7738 leader_election.cc:288] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:00.174563 7418 leader_election.cc:302] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168; no voters:
I20240502 21:27:00.175328 7832 raft_consensus.cc:2781] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:00.188640 7883 tablet_replica.cc:330] T 319b287b9c05456882890f0bb4cd1d35 P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:27:00.189002 7883 raft_consensus.cc:2229] T 319b287b9c05456882890f0bb4cd1d35 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:00.189407 7883 raft_consensus.cc:2258] T 319b287b9c05456882890f0bb4cd1d35 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:00.191684 7883 ts_tablet_manager.cc:1875] T 319b287b9c05456882890f0bb4cd1d35 P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 21:27:00.194644 4753 tablet.cc:2369] T ba5b4c81211647dd8383ec9484bb9c75 P 398c3c8f3195492c900439ad5e1931bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:27:00.201208 7883 ts_tablet_manager.cc:1888] T 319b287b9c05456882890f0bb4cd1d35 P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:00.206225 7883 log.cc:1196] T 319b287b9c05456882890f0bb4cd1d35 P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/319b287b9c05456882890f0bb4cd1d35
I20240502 21:27:00.206914 7883 ts_tablet_manager.cc:1909] T 319b287b9c05456882890f0bb4cd1d35 P 39981a8dfe94462b9a73ba489af4c714: Deleting consensus metadata
I20240502 21:27:00.208591 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet 319b287b9c05456882890f0bb4cd1d35 (table rebalancer_dynamic_table_new_name [id=7a9b82ace20943c99c2d9d15fd3a73c8]) successfully deleted
I20240502 21:27:00.228065 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba5b4c81211647dd8383ec9484bb9c75" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:27:00.228408 4705 raft_consensus.cc:2454] T ba5b4c81211647dd8383ec9484bb9c75 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:00.228904 4704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba5b4c81211647dd8383ec9484bb9c75" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:27:00.229168 4704 raft_consensus.cc:3037] T ba5b4c81211647dd8383ec9484bb9c75 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.233610 4704 raft_consensus.cc:2454] T ba5b4c81211647dd8383ec9484bb9c75 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:00.250545 7884 tablet_replica.cc:330] T 319b287b9c05456882890f0bb4cd1d35 P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:27:00.251412 7884 raft_consensus.cc:2229] T 319b287b9c05456882890f0bb4cd1d35 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:00.251969 7884 raft_consensus.cc:2258] T 319b287b9c05456882890f0bb4cd1d35 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:00.257763 7832 raft_consensus.cc:686] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Becoming Leader. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Running, Role: LEADER
I20240502 21:27:00.258277 7832 consensus_queue.cc:237] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:00.258960 4300 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:00.264364 7884 ts_tablet_manager.cc:1875] T 319b287b9c05456882890f0bb4cd1d35 P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:00.264991 7882 tablet_replica.cc:330] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04: stopping tablet replica
I20240502 21:27:00.265542 7882 raft_consensus.cc:2229] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:00.267370 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d095a5e8f1f34b5ea2b6797ef06f79c6" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:00.267773 7618 raft_consensus.cc:2454] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 0.
I20240502 21:27:00.269649 7463 tablet_service.cc:1467] Processing CreateTablet for tablet 079f56a589e348f788a957339f6985a2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c82646dcfc8446e18f496dd05b531e65]), partition=
I20240502 21:27:00.270040 7463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 079f56a589e348f788a957339f6985a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.270251 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 079f56a589e348f788a957339f6985a2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c82646dcfc8446e18f496dd05b531e65]), partition=
I20240502 21:27:00.270659 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 079f56a589e348f788a957339f6985a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.268411 7282 leader_election.cc:302] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:00.273651 7738 raft_consensus.cc:2781] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:00.273775 7738 raft_consensus.cc:482] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:00.273854 7738 raft_consensus.cc:3037] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.276559 4300 catalog_manager.cc:5484] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a72286397c4edc9a63eca3412be168 (127.29.82.71). New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:00.276746 7738 raft_consensus.cc:504] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:00.277498 7738 leader_election.cc:288] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Requested vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:00.276880 7595 tablet_service.cc:1467] Processing CreateTablet for tablet 079f56a589e348f788a957339f6985a2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c82646dcfc8446e18f496dd05b531e65]), partition=
I20240502 21:27:00.278606 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d095a5e8f1f34b5ea2b6797ef06f79c6" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:00.278806 7865 tablet_bootstrap.cc:492] T 079f56a589e348f788a957339f6985a2 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:00.278864 7618 raft_consensus.cc:3037] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.280349 7865 tablet_bootstrap.cc:654] T 079f56a589e348f788a957339f6985a2 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.280486 7595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 079f56a589e348f788a957339f6985a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.281697 7618 raft_consensus.cc:2454] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
I20240502 21:27:00.282308 7864 raft_consensus.cc:482] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:00.282500 7864 raft_consensus.cc:504] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:00.283452 7864 leader_election.cc:288] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:00.283630 7857 tablet_bootstrap.cc:492] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:00.284106 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d095a5e8f1f34b5ea2b6797ef06f79c6" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:00.284448 7350 raft_consensus.cc:2379] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39981a8dfe94462b9a73ba489af4c714 in current term 1: Already voted for candidate d0456c5e55a542128688904e003d403c in this term.
I20240502 21:27:00.285327 7857 tablet_bootstrap.cc:654] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.286135 7282 leader_election.cc:302] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:00.286478 7738 raft_consensus.cc:2781] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:00.293820 7882 raft_consensus.cc:2258] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:00.296145 7882 ts_tablet_manager.cc:1875] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:00.304315 7857 tablet_bootstrap.cc:492] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:00.304566 7857 ts_tablet_manager.cc:1367] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.008s
I20240502 21:27:00.306691 7857 raft_consensus.cc:348] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:00.306954 7857 raft_consensus.cc:374] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.307077 7857 raft_consensus.cc:729] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.307425 7857 consensus_queue.cc:260] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:00.308120 7857 ts_tablet_manager.cc:1398] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:00.311750 7738 raft_consensus.cc:686] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Becoming Leader. State: Replica: d0456c5e55a542128688904e003d403c, State: Running, Role: LEADER
I20240502 21:27:00.313302 7738 consensus_queue.cc:237] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:00.301184 7858 tablet_bootstrap.cc:492] T 079f56a589e348f788a957339f6985a2 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:00.316671 7858 tablet_bootstrap.cc:654] T 079f56a589e348f788a957339f6985a2 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.317361 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d095a5e8f1f34b5ea2b6797ef06f79c6" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:00.317651 7214 raft_consensus.cc:2454] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 0.
I20240502 21:27:00.318046 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d095a5e8f1f34b5ea2b6797ef06f79c6" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:00.318243 7215 raft_consensus.cc:3037] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.321580 7215 raft_consensus.cc:2454] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
I20240502 21:27:00.321663 7884 ts_tablet_manager.cc:1888] T 319b287b9c05456882890f0bb4cd1d35 P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:00.321823 7884 log.cc:1196] T 319b287b9c05456882890f0bb4cd1d35 P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/319b287b9c05456882890f0bb4cd1d35
I20240502 21:27:00.322427 7884 ts_tablet_manager.cc:1909] T 319b287b9c05456882890f0bb4cd1d35 P d0456c5e55a542128688904e003d403c: Deleting consensus metadata
I20240502 21:27:00.324304 7865 tablet_bootstrap.cc:492] T 079f56a589e348f788a957339f6985a2 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:27:00.324237 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet 319b287b9c05456882890f0bb4cd1d35 (table rebalancer_dynamic_table_new_name [id=7a9b82ace20943c99c2d9d15fd3a73c8]) successfully deleted
I20240502 21:27:00.326102 7865 ts_tablet_manager.cc:1367] T 079f56a589e348f788a957339f6985a2 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.047s user 0.014s sys 0.009s
I20240502 21:27:00.328302 7865 raft_consensus.cc:348] T 079f56a589e348f788a957339f6985a2 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:00.328604 7865 raft_consensus.cc:374] T 079f56a589e348f788a957339f6985a2 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.328713 7865 raft_consensus.cc:729] T 079f56a589e348f788a957339f6985a2 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.329094 7865 consensus_queue.cc:260] T 079f56a589e348f788a957339f6985a2 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:00.329856 7865 ts_tablet_manager.cc:1398] T 079f56a589e348f788a957339f6985a2 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:27:00.331050 4292 catalog_manager.cc:5484] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c reported cstate change: term changed from 0 to 1, leader changed from <none> to d0456c5e55a542128688904e003d403c (127.29.82.70). New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:00.336140 7832 raft_consensus.cc:482] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:00.336324 7832 raft_consensus.cc:504] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:00.336937 7832 leader_election.cc:288] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:00.337816 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f800bfd495a94f2a952bc57a348304d0" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:00.338063 7618 raft_consensus.cc:2454] T f800bfd495a94f2a952bc57a348304d0 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:00.338214 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f800bfd495a94f2a952bc57a348304d0" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:00.338457 7215 raft_consensus.cc:2454] T f800bfd495a94f2a952bc57a348304d0 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:00.338586 7418 leader_election.cc:302] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168; no voters:
I20240502 21:27:00.338951 7832 raft_consensus.cc:2781] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:00.339051 7832 raft_consensus.cc:482] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:00.339154 7832 raft_consensus.cc:3037] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.341046 7832 raft_consensus.cc:504] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:00.341573 7832 leader_election.cc:288] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Requested vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:00.341948 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f800bfd495a94f2a952bc57a348304d0" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:00.342135 7215 raft_consensus.cc:3037] T f800bfd495a94f2a952bc57a348304d0 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.342267 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f800bfd495a94f2a952bc57a348304d0" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:00.342448 7618 raft_consensus.cc:3037] T f800bfd495a94f2a952bc57a348304d0 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.345038 7215 raft_consensus.cc:2454] T f800bfd495a94f2a952bc57a348304d0 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:00.345562 7419 leader_election.cc:302] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:00.345877 7832 raft_consensus.cc:2781] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:00.345937 7618 raft_consensus.cc:2454] T f800bfd495a94f2a952bc57a348304d0 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:00.346087 7832 raft_consensus.cc:686] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Becoming Leader. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Running, Role: LEADER
I20240502 21:27:00.346427 7832 consensus_queue.cc:237] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:00.352824 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d095a5e8f1f34b5ea2b6797ef06f79c6" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:00.353358 7618 raft_consensus.cc:2379] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39981a8dfe94462b9a73ba489af4c714 in current term 1: Already voted for candidate d0456c5e55a542128688904e003d403c in this term.
I20240502 21:27:00.354002 7149 leader_election.cc:302] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714; no voters: 0ddd486109f5434cace13d60d633ac04, d0456c5e55a542128688904e003d403c
I20240502 21:27:00.354406 7864 raft_consensus.cc:2726] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:27:00.368081 7858 tablet_bootstrap.cc:492] T 079f56a589e348f788a957339f6985a2 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:00.368288 4292 catalog_manager.cc:5484] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a72286397c4edc9a63eca3412be168 (127.29.82.71). New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:27:00.368978 7858 ts_tablet_manager.cc:1367] T 079f56a589e348f788a957339f6985a2 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.068s user 0.003s sys 0.009s
I20240502 21:27:00.370321 7858 raft_consensus.cc:348] T 079f56a589e348f788a957339f6985a2 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:00.370627 7858 raft_consensus.cc:374] T 079f56a589e348f788a957339f6985a2 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.370747 7858 raft_consensus.cc:729] T 079f56a589e348f788a957339f6985a2 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.371062 7858 consensus_queue.cc:260] T 079f56a589e348f788a957339f6985a2 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:00.372803 7858 ts_tablet_manager.cc:1398] T 079f56a589e348f788a957339f6985a2 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:00.376506 7882 ts_tablet_manager.cc:1888] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:00.376719 7882 log.cc:1196] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/wals/319b287b9c05456882890f0bb4cd1d35
I20240502 21:27:00.377337 7882 ts_tablet_manager.cc:1909] T 319b287b9c05456882890f0bb4cd1d35 P 0ddd486109f5434cace13d60d633ac04: Deleting consensus metadata
I20240502 21:27:00.387785 4285 catalog_manager.cc:4830] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): tablet 319b287b9c05456882890f0bb4cd1d35 (table rebalancer_dynamic_table_new_name [id=7a9b82ace20943c99c2d9d15fd3a73c8]) successfully deleted
I20240502 21:27:00.405615 4292 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240502 21:27:00.405903 4292 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240502 21:27:00.410048 4292 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet c856714eb4ea454d818d467d84757eb1
I20240502 21:27:00.411463 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet c856714eb4ea454d818d467d84757eb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:00 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:00.411801 7716 tablet_replica.cc:330] T c856714eb4ea454d818d467d84757eb1 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:00.412025 7716 raft_consensus.cc:2229] T c856714eb4ea454d818d467d84757eb1 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:00.412320 7716 raft_consensus.cc:2258] T c856714eb4ea454d818d467d84757eb1 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:00.413067 7716 ts_tablet_manager.cc:1875] T c856714eb4ea454d818d467d84757eb1 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:00.416985 4817 tablet_service.cc:1514] Processing DeleteTablet for tablet c856714eb4ea454d818d467d84757eb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:00 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:27:00.417615 7718 tablet_replica.cc:330] T c856714eb4ea454d818d467d84757eb1 P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:27:00.417896 7718 raft_consensus.cc:2229] T c856714eb4ea454d818d467d84757eb1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:00.418129 7718 raft_consensus.cc:2258] T c856714eb4ea454d818d467d84757eb1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:00.418574 7716 ts_tablet_manager.cc:1888] T c856714eb4ea454d818d467d84757eb1 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:00.418716 7716 log.cc:1196] T c856714eb4ea454d818d467d84757eb1 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/c856714eb4ea454d818d467d84757eb1
I20240502 21:27:00.418908 7718 ts_tablet_manager.cc:1875] T c856714eb4ea454d818d467d84757eb1 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:00.419245 7716 ts_tablet_manager.cc:1909] T c856714eb4ea454d818d467d84757eb1 P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:00.420354 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet c856714eb4ea454d818d467d84757eb1 (table rebalance_test_table_6 [id=4cf0231801934a10bad256a243b59d8a]) successfully deleted
I20240502 21:27:00.421311 4292 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 06fa2015dc7b4d97bf7e00833e5fa4b6
I20240502 21:27:00.421504 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet c856714eb4ea454d818d467d84757eb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:00 UTC) from {username='slave'} at 127.0.0.1:60018
I20240502 21:27:00.421846 7717 tablet_replica.cc:330] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:27:00.422107 7717 raft_consensus.cc:2229] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:00.422313 4292 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet ccedd81bdfda43fbac7df75535af5cb6
I20240502 21:27:00.422358 4817 tablet_service.cc:1514] Processing DeleteTablet for tablet 06fa2015dc7b4d97bf7e00833e5fa4b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:00 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:27:00.422511 7717 raft_consensus.cc:2258] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:00.423311 7717 ts_tablet_manager.cc:1875] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:00.423805 4817 tablet_service.cc:1514] Processing DeleteTablet for tablet ccedd81bdfda43fbac7df75535af5cb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:00 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:27:00.424965 7718 ts_tablet_manager.cc:1888] T c856714eb4ea454d818d467d84757eb1 P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:00.425067 7718 log.cc:1196] T c856714eb4ea454d818d467d84757eb1 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/c856714eb4ea454d818d467d84757eb1
I20240502 21:27:00.425387 7718 ts_tablet_manager.cc:1909] T c856714eb4ea454d818d467d84757eb1 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:27:00.426142 7718 tablet_replica.cc:330] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:27:00.426244 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet c856714eb4ea454d818d467d84757eb1 (table rebalance_test_table_6 [id=4cf0231801934a10bad256a243b59d8a]) successfully deleted
I20240502 21:27:00.426348 7718 raft_consensus.cc:2229] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:00.426573 7718 raft_consensus.cc:2258] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:00.427132 7718 ts_tablet_manager.cc:1875] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:00.427731 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet 06fa2015dc7b4d97bf7e00833e5fa4b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:00 UTC) from {username='slave'} at 127.0.0.1:44764
I20240502 21:27:00.428375 7897 tablet_replica.cc:330] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:27:00.428515 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet 06fa2015dc7b4d97bf7e00833e5fa4b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:00 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:00.428674 7897 raft_consensus.cc:2229] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:00.428711 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet ccedd81bdfda43fbac7df75535af5cb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:00 UTC) from {username='slave'} at 127.0.0.1:60018
I20240502 21:27:00.428795 4422 tablet_service.cc:1514] Processing DeleteTablet for tablet ccedd81bdfda43fbac7df75535af5cb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:00 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:00.428972 7897 raft_consensus.cc:2258] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:00.429720 7897 ts_tablet_manager.cc:1875] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:00.431798 7717 ts_tablet_manager.cc:1888] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:00.432008 7717 log.cc:1196] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/c856714eb4ea454d818d467d84757eb1
I20240502 21:27:00.432574 7718 ts_tablet_manager.cc:1888] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:00.432767 7718 log.cc:1196] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/06fa2015dc7b4d97bf7e00833e5fa4b6
I20240502 21:27:00.433319 7718 ts_tablet_manager.cc:1909] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:27:00.433562 7717 ts_tablet_manager.cc:1909] T c856714eb4ea454d818d467d84757eb1 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting consensus metadata
I20240502 21:27:00.434388 7718 tablet_replica.cc:330] T ccedd81bdfda43fbac7df75535af5cb6 P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:27:00.434428 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet 06fa2015dc7b4d97bf7e00833e5fa4b6 (table rebalance_test_table_6 [id=4cf0231801934a10bad256a243b59d8a]) successfully deleted
I20240502 21:27:00.434692 7718 raft_consensus.cc:2229] T ccedd81bdfda43fbac7df75535af5cb6 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:00.435063 7718 raft_consensus.cc:2258] T ccedd81bdfda43fbac7df75535af5cb6 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:00.435770 7718 ts_tablet_manager.cc:1875] T ccedd81bdfda43fbac7df75535af5cb6 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:00.436725 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet c856714eb4ea454d818d467d84757eb1 (table rebalance_test_table_6 [id=4cf0231801934a10bad256a243b59d8a]) successfully deleted
I20240502 21:27:00.437032 7716 tablet_replica.cc:330] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:00.437269 7716 raft_consensus.cc:2229] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:00.437639 7716 raft_consensus.cc:2258] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:00.437814 7897 ts_tablet_manager.cc:1888] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:00.437945 7897 log.cc:1196] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/06fa2015dc7b4d97bf7e00833e5fa4b6
I20240502 21:27:00.438428 7897 ts_tablet_manager.cc:1909] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 398c3c8f3195492c900439ad5e1931bb: Deleting consensus metadata
I20240502 21:27:00.438503 7716 ts_tablet_manager.cc:1875] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:00.439859 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet 06fa2015dc7b4d97bf7e00833e5fa4b6 (table rebalance_test_table_6 [id=4cf0231801934a10bad256a243b59d8a]) successfully deleted
I20240502 21:27:00.440070 7717 tablet_replica.cc:330] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:27:00.440299 7717 raft_consensus.cc:2229] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:00.440424 7718 ts_tablet_manager.cc:1888] T ccedd81bdfda43fbac7df75535af5cb6 P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:00.440558 7718 log.cc:1196] T ccedd81bdfda43fbac7df75535af5cb6 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/ccedd81bdfda43fbac7df75535af5cb6
I20240502 21:27:00.440596 7717 raft_consensus.cc:2258] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:00.440996 7718 ts_tablet_manager.cc:1909] T ccedd81bdfda43fbac7df75535af5cb6 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:27:00.441309 7717 ts_tablet_manager.cc:1875] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:00.441918 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet ccedd81bdfda43fbac7df75535af5cb6 (table rebalance_test_table_6 [id=4cf0231801934a10bad256a243b59d8a]) successfully deleted
I20240502 21:27:00.445572 7717 ts_tablet_manager.cc:1888] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:00.445701 7717 log.cc:1196] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/ccedd81bdfda43fbac7df75535af5cb6
I20240502 21:27:00.446069 7717 ts_tablet_manager.cc:1909] T ccedd81bdfda43fbac7df75535af5cb6 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting consensus metadata
I20240502 21:27:00.446913 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet ccedd81bdfda43fbac7df75535af5cb6 (table rebalance_test_table_6 [id=4cf0231801934a10bad256a243b59d8a]) successfully deleted
I20240502 21:27:00.450873 7716 ts_tablet_manager.cc:1888] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:00.451012 7716 log.cc:1196] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/06fa2015dc7b4d97bf7e00833e5fa4b6
I20240502 21:27:00.451377 7716 ts_tablet_manager.cc:1909] T 06fa2015dc7b4d97bf7e00833e5fa4b6 P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:00.452230 7716 tablet_replica.cc:330] T ccedd81bdfda43fbac7df75535af5cb6 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:00.452350 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet 06fa2015dc7b4d97bf7e00833e5fa4b6 (table rebalance_test_table_6 [id=4cf0231801934a10bad256a243b59d8a]) successfully deleted
I20240502 21:27:00.452422 7716 raft_consensus.cc:2229] T ccedd81bdfda43fbac7df75535af5cb6 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:00.452562 7832 raft_consensus.cc:482] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:00.452684 7716 raft_consensus.cc:2258] T ccedd81bdfda43fbac7df75535af5cb6 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:00.452728 7832 raft_consensus.cc:504] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:00.453404 7716 ts_tablet_manager.cc:1875] T ccedd81bdfda43fbac7df75535af5cb6 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:00.453563 7832 leader_election.cc:288] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:00.454066 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "079f56a589e348f788a957339f6985a2" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:00.454200 4704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "079f56a589e348f788a957339f6985a2" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:27:00.454363 7618 raft_consensus.cc:2454] T 079f56a589e348f788a957339f6985a2 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:00.454444 4704 raft_consensus.cc:2454] T 079f56a589e348f788a957339f6985a2 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:00.454821 7418 leader_election.cc:302] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168; no voters:
I20240502 21:27:00.455133 7832 raft_consensus.cc:2781] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:00.455256 7832 raft_consensus.cc:482] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:00.455335 7832 raft_consensus.cc:3037] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.458088 7832 raft_consensus.cc:504] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:00.458767 7832 leader_election.cc:288] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:00.459007 7716 ts_tablet_manager.cc:1888] T ccedd81bdfda43fbac7df75535af5cb6 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:00.459120 7716 log.cc:1196] T ccedd81bdfda43fbac7df75535af5cb6 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/ccedd81bdfda43fbac7df75535af5cb6
I20240502 21:27:00.459146 4704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "079f56a589e348f788a957339f6985a2" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:27:00.459357 4704 raft_consensus.cc:3037] T 079f56a589e348f788a957339f6985a2 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.459280 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "079f56a589e348f788a957339f6985a2" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:00.459522 7618 raft_consensus.cc:3037] T 079f56a589e348f788a957339f6985a2 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.459652 7716 ts_tablet_manager.cc:1909] T ccedd81bdfda43fbac7df75535af5cb6 P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:00.460695 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet ccedd81bdfda43fbac7df75535af5cb6 (table rebalance_test_table_6 [id=4cf0231801934a10bad256a243b59d8a]) successfully deleted
I20240502 21:27:00.461542 4704 raft_consensus.cc:2454] T 079f56a589e348f788a957339f6985a2 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:00.461946 7420 leader_election.cc:302] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 398c3c8f3195492c900439ad5e1931bb; no voters:
I20240502 21:27:00.462116 7618 raft_consensus.cc:2454] T 079f56a589e348f788a957339f6985a2 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:00.462220 7832 raft_consensus.cc:2781] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:00.462388 7832 raft_consensus.cc:686] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Becoming Leader. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Running, Role: LEADER
I20240502 21:27:00.462656 7832 consensus_queue.cc:237] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:00.465390 4294 catalog_manager.cc:5484] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a72286397c4edc9a63eca3412be168 (127.29.82.71). New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:00.580711 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:00.586850 4704 raft_consensus.cc:1261] T 079f56a589e348f788a957339f6985a2 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:00.587324 7618 raft_consensus.cc:1261] T 079f56a589e348f788a957339f6985a2 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:00.588141 7888 consensus_queue.cc:1035] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.588423 7832 consensus_queue.cc:1035] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.605002 4294 catalog_manager.cc:5576] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537) does not have the latest schema for tablet 079f56a589e348f788a957339f6985a2 (table rebalancer_dynamic_table [id=c82646dcfc8446e18f496dd05b531e65]). Expected version 1 got 0
I20240502 21:27:00.605970 4300 catalog_manager.cc:5576] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223) does not have the latest schema for tablet 079f56a589e348f788a957339f6985a2 (table rebalancer_dynamic_table [id=c82646dcfc8446e18f496dd05b531e65]). Expected version 1 got 0
I20240502 21:27:00.607337 7904 tablet.cc:1722] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:00.608978 7906 tablet.cc:1722] T 079f56a589e348f788a957339f6985a2 P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:00.609202 7904 tablet.cc:1620] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:00.609644 7906 tablet.cc:1620] T 079f56a589e348f788a957339f6985a2 P 0ddd486109f5434cace13d60d633ac04: MemRowSet was empty: no flush needed.
I20240502 21:27:00.609655 7905 tablet.cc:1722] T 079f56a589e348f788a957339f6985a2 P 398c3c8f3195492c900439ad5e1931bb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:00.611150 4300 catalog_manager.cc:5576] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537) does not have the latest schema for tablet 079f56a589e348f788a957339f6985a2 (table rebalancer_dynamic_table [id=c82646dcfc8446e18f496dd05b531e65]). Expected version 1 got 0
I20240502 21:27:00.611425 7905 tablet.cc:1620] T 079f56a589e348f788a957339f6985a2 P 398c3c8f3195492c900439ad5e1931bb: MemRowSet was empty: no flush needed.
I20240502 21:27:00.621310 4294 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=c82646dcfc8446e18f496dd05b531e65] alter complete (version 1)
I20240502 21:27:00.725975 7738 consensus_queue.cc:1035] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.729207 4292 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:00.735294 7909 consensus_queue.cc:1035] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.739794 7906 tablet.cc:1722] T 079f56a589e348f788a957339f6985a2 P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:00.739902 7904 tablet.cc:1722] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:00.740396 7904 tablet.cc:1620] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:00.740593 7905 tablet.cc:1722] T 079f56a589e348f788a957339f6985a2 P 398c3c8f3195492c900439ad5e1931bb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:00.741412 7906 tablet.cc:1620] T 079f56a589e348f788a957339f6985a2 P 0ddd486109f5434cace13d60d633ac04: MemRowSet was empty: no flush needed.
I20240502 21:27:00.741812 7905 tablet.cc:1620] T 079f56a589e348f788a957339f6985a2 P 398c3c8f3195492c900439ad5e1931bb: MemRowSet was empty: no flush needed.
I20240502 21:27:00.749202 4292 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=c82646dcfc8446e18f496dd05b531e65] alter complete (version 2)
I20240502 21:27:00.799427 7888 consensus_queue.cc:1035] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.805248 7832 consensus_queue.cc:1035] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.857960 4292 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:00.865964 7904 tablet.cc:1722] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:00.868106 7906 tablet.cc:1722] T 079f56a589e348f788a957339f6985a2 P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:00.868516 7905 tablet.cc:1722] T 079f56a589e348f788a957339f6985a2 P 398c3c8f3195492c900439ad5e1931bb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:00.874647 4292 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=c82646dcfc8446e18f496dd05b531e65] alter complete (version 3)
I20240502 21:27:00.882081 7832 consensus_queue.cc:1035] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.886379 7832 consensus_queue.cc:1035] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.929142 4300 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240502 21:27:00.929436 4300 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240502 21:27:00.933244 4300 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 8f9c10843d63429fa2643e12050b9f29
I20240502 21:27:00.933737 4300 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet dad7605b83064a3abebd26b3b6570f27
I20240502 21:27:00.934274 4300 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 350666c5e39744ff91d5adc8a6f2977d
I20240502 21:27:00.934309 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f9c10843d63429fa2643e12050b9f29 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:00 UTC) from {username='slave'} at 127.0.0.1:60018
I20240502 21:27:00.934505 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f9c10843d63429fa2643e12050b9f29 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:00 UTC) from {username='slave'} at 127.0.0.1:44764
I20240502 21:27:00.934641 7717 tablet_replica.cc:330] T 8f9c10843d63429fa2643e12050b9f29 P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:27:00.934657 4422 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f9c10843d63429fa2643e12050b9f29 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:00 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:00.934779 7897 tablet_replica.cc:330] T 8f9c10843d63429fa2643e12050b9f29 P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:27:00.934877 7717 raft_consensus.cc:2229] T 8f9c10843d63429fa2643e12050b9f29 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:00.934949 7716 tablet_replica.cc:330] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:00.935024 7897 raft_consensus.cc:2229] T 8f9c10843d63429fa2643e12050b9f29 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:00.935163 7717 raft_consensus.cc:2258] T 8f9c10843d63429fa2643e12050b9f29 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:00.935180 7716 raft_consensus.cc:2229] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:00.935312 7897 raft_consensus.cc:2258] T 8f9c10843d63429fa2643e12050b9f29 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:00.935480 7716 raft_consensus.cc:2258] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:00.935923 4817 tablet_service.cc:1514] Processing DeleteTablet for tablet dad7605b83064a3abebd26b3b6570f27 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:00 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:27:00.935971 7717 ts_tablet_manager.cc:1875] T 8f9c10843d63429fa2643e12050b9f29 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:00.936071 7897 ts_tablet_manager.cc:1875] T 8f9c10843d63429fa2643e12050b9f29 P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:00.936511 7716 ts_tablet_manager.cc:1875] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:00.939749 7718 tablet_replica.cc:330] T dad7605b83064a3abebd26b3b6570f27 P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:27:00.939961 7718 raft_consensus.cc:2229] T dad7605b83064a3abebd26b3b6570f27 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:00.940238 7718 raft_consensus.cc:2258] T dad7605b83064a3abebd26b3b6570f27 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:00.940326 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet dad7605b83064a3abebd26b3b6570f27 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:00 UTC) from {username='slave'} at 127.0.0.1:60018
I20240502 21:27:00.940577 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet 350666c5e39744ff91d5adc8a6f2977d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:00 UTC) from {username='slave'} at 127.0.0.1:44764
I20240502 21:27:00.940886 7718 ts_tablet_manager.cc:1875] T dad7605b83064a3abebd26b3b6570f27 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:00.942644 7717 ts_tablet_manager.cc:1888] T 8f9c10843d63429fa2643e12050b9f29 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:00.942773 7717 log.cc:1196] T 8f9c10843d63429fa2643e12050b9f29 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/8f9c10843d63429fa2643e12050b9f29
I20240502 21:27:00.943189 7717 ts_tablet_manager.cc:1909] T 8f9c10843d63429fa2643e12050b9f29 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting consensus metadata
I20240502 21:27:00.944011 7897 ts_tablet_manager.cc:1888] T 8f9c10843d63429fa2643e12050b9f29 P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:00.944010 4422 tablet_service.cc:1514] Processing DeleteTablet for tablet dad7605b83064a3abebd26b3b6570f27 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:00 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:00.944178 7897 log.cc:1196] T 8f9c10843d63429fa2643e12050b9f29 P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/8f9c10843d63429fa2643e12050b9f29
I20240502 21:27:00.944695 7717 tablet_replica.cc:330] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:27:00.944828 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet 8f9c10843d63429fa2643e12050b9f29 (table rebalance_test_table_8 [id=e9b2eb99277d4f6a97dca65e69292ffd]) successfully deleted
I20240502 21:27:00.944871 4817 tablet_service.cc:1514] Processing DeleteTablet for tablet 350666c5e39744ff91d5adc8a6f2977d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:00 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:27:00.945116 7717 raft_consensus.cc:2229] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:00.945323 4422 tablet_service.cc:1514] Processing DeleteTablet for tablet 350666c5e39744ff91d5adc8a6f2977d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:00 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:00.945484 7717 raft_consensus.cc:2258] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:00.945981 7897 ts_tablet_manager.cc:1909] T 8f9c10843d63429fa2643e12050b9f29 P 398c3c8f3195492c900439ad5e1931bb: Deleting consensus metadata
I20240502 21:27:00.946300 7717 ts_tablet_manager.cc:1875] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:00.947054 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet 8f9c10843d63429fa2643e12050b9f29 (table rebalance_test_table_8 [id=e9b2eb99277d4f6a97dca65e69292ffd]) successfully deleted
I20240502 21:27:00.947428 7897 tablet_replica.cc:330] T 350666c5e39744ff91d5adc8a6f2977d P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:27:00.947739 7897 raft_consensus.cc:2229] T 350666c5e39744ff91d5adc8a6f2977d P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:00.947846 7716 ts_tablet_manager.cc:1888] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:00.947990 7897 raft_consensus.cc:2258] T 350666c5e39744ff91d5adc8a6f2977d P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:00.948037 7716 log.cc:1196] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/8f9c10843d63429fa2643e12050b9f29
I20240502 21:27:00.948684 7716 ts_tablet_manager.cc:1909] T 8f9c10843d63429fa2643e12050b9f29 P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:00.948722 7718 ts_tablet_manager.cc:1888] T dad7605b83064a3abebd26b3b6570f27 P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:00.948913 7718 log.cc:1196] T dad7605b83064a3abebd26b3b6570f27 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/dad7605b83064a3abebd26b3b6570f27
I20240502 21:27:00.949128 7897 ts_tablet_manager.cc:1875] T 350666c5e39744ff91d5adc8a6f2977d P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:00.949803 7716 tablet_replica.cc:330] T dad7605b83064a3abebd26b3b6570f27 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:00.951056 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet 8f9c10843d63429fa2643e12050b9f29 (table rebalance_test_table_8 [id=e9b2eb99277d4f6a97dca65e69292ffd]) successfully deleted
I20240502 21:27:00.951767 7718 ts_tablet_manager.cc:1909] T dad7605b83064a3abebd26b3b6570f27 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:27:00.952808 7718 tablet_replica.cc:330] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:27:00.953038 7717 ts_tablet_manager.cc:1888] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:00.953099 7718 raft_consensus.cc:2229] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:00.953224 7717 log.cc:1196] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/dad7605b83064a3abebd26b3b6570f27
I20240502 21:27:00.953524 7718 raft_consensus.cc:2258] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:00.953711 7717 ts_tablet_manager.cc:1909] T dad7605b83064a3abebd26b3b6570f27 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting consensus metadata
I20240502 21:27:00.954277 7718 ts_tablet_manager.cc:1875] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:00.954684 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet dad7605b83064a3abebd26b3b6570f27 (table rebalance_test_table_8 [id=e9b2eb99277d4f6a97dca65e69292ffd]) successfully deleted
I20240502 21:27:00.955003 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet dad7605b83064a3abebd26b3b6570f27 (table rebalance_test_table_8 [id=e9b2eb99277d4f6a97dca65e69292ffd]) successfully deleted
I20240502 21:27:00.955027 7716 raft_consensus.cc:2229] T dad7605b83064a3abebd26b3b6570f27 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:00.955271 7716 raft_consensus.cc:2258] T dad7605b83064a3abebd26b3b6570f27 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:00.955596 7897 ts_tablet_manager.cc:1888] T 350666c5e39744ff91d5adc8a6f2977d P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:00.950266 4300 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:00.955963 7716 ts_tablet_manager.cc:1875] T dad7605b83064a3abebd26b3b6570f27 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:00.956023 7897 log.cc:1196] T 350666c5e39744ff91d5adc8a6f2977d P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/350666c5e39744ff91d5adc8a6f2977d
I20240502 21:27:00.956575 7897 ts_tablet_manager.cc:1909] T 350666c5e39744ff91d5adc8a6f2977d P 398c3c8f3195492c900439ad5e1931bb: Deleting consensus metadata
I20240502 21:27:00.957756 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet 350666c5e39744ff91d5adc8a6f2977d (table rebalance_test_table_8 [id=e9b2eb99277d4f6a97dca65e69292ffd]) successfully deleted
I20240502 21:27:00.963075 7716 ts_tablet_manager.cc:1888] T dad7605b83064a3abebd26b3b6570f27 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:00.963234 7716 log.cc:1196] T dad7605b83064a3abebd26b3b6570f27 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/dad7605b83064a3abebd26b3b6570f27
I20240502 21:27:00.963685 7716 ts_tablet_manager.cc:1909] T dad7605b83064a3abebd26b3b6570f27 P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:00.964568 7716 tablet_replica.cc:330] T 350666c5e39744ff91d5adc8a6f2977d P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:00.964583 7718 ts_tablet_manager.cc:1888] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:00.964730 7718 log.cc:1196] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/350666c5e39744ff91d5adc8a6f2977d
I20240502 21:27:00.964812 7716 raft_consensus.cc:2229] T 350666c5e39744ff91d5adc8a6f2977d P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:00.964810 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet dad7605b83064a3abebd26b3b6570f27 (table rebalance_test_table_8 [id=e9b2eb99277d4f6a97dca65e69292ffd]) successfully deleted
I20240502 21:27:00.965020 7716 raft_consensus.cc:2258] T 350666c5e39744ff91d5adc8a6f2977d P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:00.965215 7718 ts_tablet_manager.cc:1909] T 350666c5e39744ff91d5adc8a6f2977d P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:27:00.965701 7716 ts_tablet_manager.cc:1875] T 350666c5e39744ff91d5adc8a6f2977d P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:00.971153 7463 tablet_service.cc:1467] Processing CreateTablet for tablet 6d0334be884c45c3bd9e49095454dfb1 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=2ba724ea31944444b73e3c52e98f06e7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.971719 7463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d0334be884c45c3bd9e49095454dfb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.972399 7597 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3528b68bc740ce9b452356fd26bfb6 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=2ba724ea31944444b73e3c52e98f06e7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.972685 7328 tablet_service.cc:1467] Processing CreateTablet for tablet 6d0334be884c45c3bd9e49095454dfb1 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=2ba724ea31944444b73e3c52e98f06e7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.972862 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3528b68bc740ce9b452356fd26bfb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.973096 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d0334be884c45c3bd9e49095454dfb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.973515 7716 ts_tablet_manager.cc:1888] T 350666c5e39744ff91d5adc8a6f2977d P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:00.973649 7716 log.cc:1196] T 350666c5e39744ff91d5adc8a6f2977d P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/350666c5e39744ff91d5adc8a6f2977d
I20240502 21:27:00.974028 7716 ts_tablet_manager.cc:1909] T 350666c5e39744ff91d5adc8a6f2977d P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:00.975136 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet 350666c5e39744ff91d5adc8a6f2977d (table rebalance_test_table_8 [id=e9b2eb99277d4f6a97dca65e69292ffd]) successfully deleted
I20240502 21:27:00.975893 4555 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3528b68bc740ce9b452356fd26bfb6 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=2ba724ea31944444b73e3c52e98f06e7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.976349 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3528b68bc740ce9b452356fd26bfb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.972399 7595 tablet_service.cc:1467] Processing CreateTablet for tablet 6d0334be884c45c3bd9e49095454dfb1 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=2ba724ea31944444b73e3c52e98f06e7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.976771 7595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d0334be884c45c3bd9e49095454dfb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.978540 4292 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:00.978746 4292 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:00.981947 7330 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3528b68bc740ce9b452356fd26bfb6 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=2ba724ea31944444b73e3c52e98f06e7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.982524 7330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3528b68bc740ce9b452356fd26bfb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.982743 4817 tablet_service.cc:1467] Processing CreateTablet for tablet 845233761c2e45b0a23e25107f5461f1 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=2ba724ea31944444b73e3c52e98f06e7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.982792 7464 tablet_service.cc:1467] Processing CreateTablet for tablet 845233761c2e45b0a23e25107f5461f1 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=2ba724ea31944444b73e3c52e98f06e7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.983191 7464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 845233761c2e45b0a23e25107f5461f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.983193 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 845233761c2e45b0a23e25107f5461f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.987433 7329 tablet_service.cc:1467] Processing CreateTablet for tablet 845233761c2e45b0a23e25107f5461f1 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=2ba724ea31944444b73e3c52e98f06e7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.987994 7329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 845233761c2e45b0a23e25107f5461f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.988134 4292 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 079f56a589e348f788a957339f6985a2
I20240502 21:27:00.988987 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet 079f56a589e348f788a957339f6985a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:00 UTC) from {username='slave'} at 127.0.0.1:44764
I20240502 21:27:00.989272 7897 tablet_replica.cc:330] T 079f56a589e348f788a957339f6985a2 P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:27:00.989264 7596 tablet_service.cc:1514] Processing DeleteTablet for tablet 079f56a589e348f788a957339f6985a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:00 UTC) from {username='slave'} at 127.0.0.1:44444
I20240502 21:27:00.989570 7897 raft_consensus.cc:2229] T 079f56a589e348f788a957339f6985a2 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:00.989797 7897 raft_consensus.cc:2258] T 079f56a589e348f788a957339f6985a2 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:00.991748 7462 tablet_service.cc:1514] Processing DeleteTablet for tablet 079f56a589e348f788a957339f6985a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:00 UTC) from {username='slave'} at 127.0.0.1:37182
I20240502 21:27:00.992869 7897 ts_tablet_manager.cc:1875] T 079f56a589e348f788a957339f6985a2 P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:00.996896 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet 350666c5e39744ff91d5adc8a6f2977d (table rebalance_test_table_8 [id=e9b2eb99277d4f6a97dca65e69292ffd]) successfully deleted
I20240502 21:27:01.002839 7897 ts_tablet_manager.cc:1888] T 079f56a589e348f788a957339f6985a2 P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:01.003051 7897 log.cc:1196] T 079f56a589e348f788a957339f6985a2 P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/079f56a589e348f788a957339f6985a2
I20240502 21:27:01.003621 7897 ts_tablet_manager.cc:1909] T 079f56a589e348f788a957339f6985a2 P 398c3c8f3195492c900439ad5e1931bb: Deleting consensus metadata
I20240502 21:27:01.005183 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet 079f56a589e348f788a957339f6985a2 (table rebalancer_dynamic_table_new_name [id=c82646dcfc8446e18f496dd05b531e65]) successfully deleted
I20240502 21:27:01.006304 7918 tablet_bootstrap.cc:492] T 6d0334be884c45c3bd9e49095454dfb1 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:01.008775 7920 tablet_bootstrap.cc:492] T 1d3528b68bc740ce9b452356fd26bfb6 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:01.011315 7918 tablet_bootstrap.cc:654] T 6d0334be884c45c3bd9e49095454dfb1 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.013772 7920 tablet_bootstrap.cc:654] T 1d3528b68bc740ce9b452356fd26bfb6 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.031045 7923 tablet_bootstrap.cc:492] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:01.033483 7923 tablet_bootstrap.cc:654] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.035871 7922 tablet_replica.cc:330] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:27:01.036224 7925 tablet_bootstrap.cc:492] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:01.036240 7922 raft_consensus.cc:2229] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:01.036676 7922 raft_consensus.cc:2258] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:01.038640 7922 ts_tablet_manager.cc:1875] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:01.041009 7919 tablet_replica.cc:330] T 079f56a589e348f788a957339f6985a2 P 0ddd486109f5434cace13d60d633ac04: stopping tablet replica
I20240502 21:27:01.041317 7919 raft_consensus.cc:2229] T 079f56a589e348f788a957339f6985a2 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:01.041936 7919 raft_consensus.cc:2258] T 079f56a589e348f788a957339f6985a2 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:01.045233 7919 ts_tablet_manager.cc:1875] T 079f56a589e348f788a957339f6985a2 P 0ddd486109f5434cace13d60d633ac04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:01.045773 7921 tablet_bootstrap.cc:492] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:01.046233 7925 tablet_bootstrap.cc:654] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.049225 7921 tablet_bootstrap.cc:654] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.050458 7920 tablet_bootstrap.cc:492] T 1d3528b68bc740ce9b452356fd26bfb6 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:01.051175 7920 ts_tablet_manager.cc:1367] T 1d3528b68bc740ce9b452356fd26bfb6 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.015s
I20240502 21:27:01.052064 7920 raft_consensus.cc:348] T 1d3528b68bc740ce9b452356fd26bfb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:01.052417 7920 raft_consensus.cc:374] T 1d3528b68bc740ce9b452356fd26bfb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.052561 7920 raft_consensus.cc:729] T 1d3528b68bc740ce9b452356fd26bfb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.052901 7920 consensus_queue.cc:260] T 1d3528b68bc740ce9b452356fd26bfb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:01.053457 7922 ts_tablet_manager.cc:1888] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:01.054260 7923 tablet_bootstrap.cc:492] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:01.054483 7923 ts_tablet_manager.cc:1367] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20240502 21:27:01.055083 7920 ts_tablet_manager.cc:1398] T 1d3528b68bc740ce9b452356fd26bfb6 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:01.055528 7923 raft_consensus.cc:348] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:01.056984 7923 raft_consensus.cc:374] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.057096 7921 tablet_bootstrap.cc:492] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:01.057121 7923 raft_consensus.cc:729] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.057336 7921 ts_tablet_manager.cc:1367] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20240502 21:27:01.057540 7919 ts_tablet_manager.cc:1888] T 079f56a589e348f788a957339f6985a2 P 0ddd486109f5434cace13d60d633ac04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:01.057713 7919 log.cc:1196] T 079f56a589e348f788a957339f6985a2 P 0ddd486109f5434cace13d60d633ac04: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/wals/079f56a589e348f788a957339f6985a2
I20240502 21:27:01.058184 7919 ts_tablet_manager.cc:1909] T 079f56a589e348f788a957339f6985a2 P 0ddd486109f5434cace13d60d633ac04: Deleting consensus metadata
I20240502 21:27:01.058324 7921 raft_consensus.cc:348] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:01.058601 7921 raft_consensus.cc:374] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.058749 7921 raft_consensus.cc:729] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.059085 7921 consensus_queue.cc:260] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:01.059343 4285 catalog_manager.cc:4830] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): tablet 079f56a589e348f788a957339f6985a2 (table rebalancer_dynamic_table_new_name [id=c82646dcfc8446e18f496dd05b531e65]) successfully deleted
I20240502 21:27:01.059783 7921 ts_tablet_manager.cc:1398] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:01.060465 7923 consensus_queue.cc:260] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:01.061229 7923 ts_tablet_manager.cc:1398] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 21:27:01.061513 7923 tablet_bootstrap.cc:492] T 6d0334be884c45c3bd9e49095454dfb1 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:01.063184 7923 tablet_bootstrap.cc:654] T 6d0334be884c45c3bd9e49095454dfb1 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.056644 7918 tablet_bootstrap.cc:492] T 6d0334be884c45c3bd9e49095454dfb1 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:01.066401 7918 ts_tablet_manager.cc:1367] T 6d0334be884c45c3bd9e49095454dfb1 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.060s user 0.004s sys 0.013s
I20240502 21:27:01.067481 7918 raft_consensus.cc:348] T 6d0334be884c45c3bd9e49095454dfb1 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:01.067780 7918 raft_consensus.cc:374] T 6d0334be884c45c3bd9e49095454dfb1 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.067888 7918 raft_consensus.cc:729] T 6d0334be884c45c3bd9e49095454dfb1 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.068171 7918 consensus_queue.cc:260] T 6d0334be884c45c3bd9e49095454dfb1 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:01.068794 7918 ts_tablet_manager.cc:1398] T 6d0334be884c45c3bd9e49095454dfb1 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:01.069043 7918 tablet_bootstrap.cc:492] T 845233761c2e45b0a23e25107f5461f1 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:01.071076 7918 tablet_bootstrap.cc:654] T 845233761c2e45b0a23e25107f5461f1 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.072551 7922 log.cc:1196] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/079f56a589e348f788a957339f6985a2
I20240502 21:27:01.072908 7923 tablet_bootstrap.cc:492] T 6d0334be884c45c3bd9e49095454dfb1 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:01.073134 7923 ts_tablet_manager.cc:1367] T 6d0334be884c45c3bd9e49095454dfb1 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20240502 21:27:01.073343 7922 ts_tablet_manager.cc:1909] T 079f56a589e348f788a957339f6985a2 P 31a72286397c4edc9a63eca3412be168: Deleting consensus metadata
I20240502 21:27:01.074182 7923 raft_consensus.cc:348] T 6d0334be884c45c3bd9e49095454dfb1 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:01.074448 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet 079f56a589e348f788a957339f6985a2 (table rebalancer_dynamic_table_new_name [id=c82646dcfc8446e18f496dd05b531e65]) successfully deleted
I20240502 21:27:01.074666 7923 raft_consensus.cc:374] T 6d0334be884c45c3bd9e49095454dfb1 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.074757 7923 raft_consensus.cc:729] T 6d0334be884c45c3bd9e49095454dfb1 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.075017 7923 consensus_queue.cc:260] T 6d0334be884c45c3bd9e49095454dfb1 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:01.076174 7923 ts_tablet_manager.cc:1398] T 6d0334be884c45c3bd9e49095454dfb1 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:01.076608 7923 tablet_bootstrap.cc:492] T 845233761c2e45b0a23e25107f5461f1 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:01.077137 7925 tablet_bootstrap.cc:492] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:01.077347 7925 ts_tablet_manager.cc:1367] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.013s
I20240502 21:27:01.079792 7925 raft_consensus.cc:348] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:01.080065 7925 raft_consensus.cc:374] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.080160 7925 raft_consensus.cc:729] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.080415 7925 consensus_queue.cc:260] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:01.080940 7925 ts_tablet_manager.cc:1398] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:27:01.081262 7923 tablet_bootstrap.cc:654] T 845233761c2e45b0a23e25107f5461f1 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.081379 7925 tablet_bootstrap.cc:492] T 1d3528b68bc740ce9b452356fd26bfb6 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:01.083019 7925 tablet_bootstrap.cc:654] T 1d3528b68bc740ce9b452356fd26bfb6 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.086036 7918 tablet_bootstrap.cc:492] T 845233761c2e45b0a23e25107f5461f1 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:01.086227 7918 ts_tablet_manager.cc:1367] T 845233761c2e45b0a23e25107f5461f1 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.004s
I20240502 21:27:01.087383 7918 raft_consensus.cc:348] T 845233761c2e45b0a23e25107f5461f1 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:01.087816 7918 raft_consensus.cc:374] T 845233761c2e45b0a23e25107f5461f1 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.087929 7918 raft_consensus.cc:729] T 845233761c2e45b0a23e25107f5461f1 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.088358 7918 consensus_queue.cc:260] T 845233761c2e45b0a23e25107f5461f1 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:01.089200 7918 ts_tablet_manager.cc:1398] T 845233761c2e45b0a23e25107f5461f1 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:01.092263 7925 tablet_bootstrap.cc:492] T 1d3528b68bc740ce9b452356fd26bfb6 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:01.092484 7925 ts_tablet_manager.cc:1367] T 1d3528b68bc740ce9b452356fd26bfb6 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20240502 21:27:01.093334 7925 raft_consensus.cc:348] T 1d3528b68bc740ce9b452356fd26bfb6 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:01.093616 7925 raft_consensus.cc:374] T 1d3528b68bc740ce9b452356fd26bfb6 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.093734 7925 raft_consensus.cc:729] T 1d3528b68bc740ce9b452356fd26bfb6 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.093657 4294 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:01.094055 7925 consensus_queue.cc:260] T 1d3528b68bc740ce9b452356fd26bfb6 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:01.094558 7925 ts_tablet_manager.cc:1398] T 1d3528b68bc740ce9b452356fd26bfb6 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:27:01.096118 7923 tablet_bootstrap.cc:492] T 845233761c2e45b0a23e25107f5461f1 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:01.096333 7923 ts_tablet_manager.cc:1367] T 845233761c2e45b0a23e25107f5461f1 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.007s
I20240502 21:27:01.097939 7923 raft_consensus.cc:348] T 845233761c2e45b0a23e25107f5461f1 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:01.098500 7923 raft_consensus.cc:374] T 845233761c2e45b0a23e25107f5461f1 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.098599 7923 raft_consensus.cc:729] T 845233761c2e45b0a23e25107f5461f1 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.098851 7923 consensus_queue.cc:260] T 845233761c2e45b0a23e25107f5461f1 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:01.099449 7923 ts_tablet_manager.cc:1398] T 845233761c2e45b0a23e25107f5461f1 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:01.102623 7330 tablet_service.cc:1467] Processing CreateTablet for tablet 74b1b8383bc64ae39b6ffe186be7b2c8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=59592ff43c694642872233217e7c82a9]), partition=
I20240502 21:27:01.103021 7330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74b1b8383bc64ae39b6ffe186be7b2c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.103191 7194 tablet_service.cc:1467] Processing CreateTablet for tablet 74b1b8383bc64ae39b6ffe186be7b2c8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=59592ff43c694642872233217e7c82a9]), partition=
I20240502 21:27:01.103614 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74b1b8383bc64ae39b6ffe186be7b2c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.107707 7923 tablet_bootstrap.cc:492] T 74b1b8383bc64ae39b6ffe186be7b2c8 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:01.109345 7923 tablet_bootstrap.cc:654] T 74b1b8383bc64ae39b6ffe186be7b2c8 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.111073 4422 tablet_service.cc:1467] Processing CreateTablet for tablet 74b1b8383bc64ae39b6ffe186be7b2c8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=59592ff43c694642872233217e7c82a9]), partition=
I20240502 21:27:01.111055 7755 raft_consensus.cc:482] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:01.111251 7755 raft_consensus.cc:504] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:01.111459 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74b1b8383bc64ae39b6ffe186be7b2c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.112071 7923 tablet_bootstrap.cc:492] T 74b1b8383bc64ae39b6ffe186be7b2c8 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:01.112267 7923 ts_tablet_manager.cc:1367] T 74b1b8383bc64ae39b6ffe186be7b2c8 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 21:27:01.112298 7755 leader_election.cc:288] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:01.113024 7923 raft_consensus.cc:348] T 74b1b8383bc64ae39b6ffe186be7b2c8 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:01.113296 7923 raft_consensus.cc:374] T 74b1b8383bc64ae39b6ffe186be7b2c8 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.113394 7923 raft_consensus.cc:729] T 74b1b8383bc64ae39b6ffe186be7b2c8 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.113646 7923 consensus_queue.cc:260] T 74b1b8383bc64ae39b6ffe186be7b2c8 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:01.114260 7923 ts_tablet_manager.cc:1398] T 74b1b8383bc64ae39b6ffe186be7b2c8 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:01.114368 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d0334be884c45c3bd9e49095454dfb1" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:01.114617 7350 raft_consensus.cc:2454] T 6d0334be884c45c3bd9e49095454dfb1 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:01.115178 7551 leader_election.cc:302] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:01.119542 7755 raft_consensus.cc:2781] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:01.119733 7755 raft_consensus.cc:482] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:01.119850 7755 raft_consensus.cc:3037] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.123921 7755 raft_consensus.cc:504] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:01.125522 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d0334be884c45c3bd9e49095454dfb1" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:01.125839 7350 raft_consensus.cc:3037] T 6d0334be884c45c3bd9e49095454dfb1 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.129160 7941 tablet_bootstrap.cc:492] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:01.129941 7940 tablet_bootstrap.cc:492] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:01.129568 7350 raft_consensus.cc:2454] T 6d0334be884c45c3bd9e49095454dfb1 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:01.130579 7551 leader_election.cc:302] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:01.131057 7941 tablet_bootstrap.cc:654] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.131724 7940 tablet_bootstrap.cc:654] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.134310 7755 leader_election.cc:288] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Requested vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:01.136224 7755 raft_consensus.cc:2781] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:01.136533 7755 raft_consensus.cc:686] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Becoming Leader. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Running, Role: LEADER
I20240502 21:27:01.137131 7822 raft_consensus.cc:482] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:01.137027 7755 consensus_queue.cc:237] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:01.137364 7822 raft_consensus.cc:504] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:01.138512 7822 leader_election.cc:288] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:01.138897 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "845233761c2e45b0a23e25107f5461f1" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:01.139173 7484 raft_consensus.cc:2454] T 845233761c2e45b0a23e25107f5461f1 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:27:01.139824 4770 leader_election.cc:302] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:01.140929 7822 raft_consensus.cc:2781] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:01.141115 7822 raft_consensus.cc:482] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:01.141228 7822 raft_consensus.cc:3037] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.145857 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d0334be884c45c3bd9e49095454dfb1" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:01.146071 7483 raft_consensus.cc:3037] T 6d0334be884c45c3bd9e49095454dfb1 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.148865 7483 raft_consensus.cc:2454] T 6d0334be884c45c3bd9e49095454dfb1 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:01.149296 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d0334be884c45c3bd9e49095454dfb1" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:01.149535 7484 raft_consensus.cc:2362] T 6d0334be884c45c3bd9e49095454dfb1 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1. Re-sending same reply.
I20240502 21:27:01.152849 7822 raft_consensus.cc:504] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:01.155099 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "845233761c2e45b0a23e25107f5461f1" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:01.155380 7484 raft_consensus.cc:3037] T 845233761c2e45b0a23e25107f5461f1 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.158299 7941 tablet_bootstrap.cc:492] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:01.158550 7941 ts_tablet_manager.cc:1367] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.012s
I20240502 21:27:01.158479 7484 raft_consensus.cc:2454] T 845233761c2e45b0a23e25107f5461f1 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:27:01.159521 7941 raft_consensus.cc:348] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:01.159837 7941 raft_consensus.cc:374] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.159969 7941 raft_consensus.cc:729] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.160260 7941 consensus_queue.cc:260] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:01.160494 4770 leader_election.cc:302] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:01.160835 7917 raft_consensus.cc:2781] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:01.161221 7941 ts_tablet_manager.cc:1398] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:01.161325 7940 tablet_bootstrap.cc:492] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:01.161509 7940 ts_tablet_manager.cc:1367] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.032s user 0.019s sys 0.008s
I20240502 21:27:01.162515 7940 raft_consensus.cc:348] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:01.162772 7940 raft_consensus.cc:374] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.162894 7940 raft_consensus.cc:729] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.163169 7940 consensus_queue.cc:260] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:01.164875 7940 ts_tablet_manager.cc:1398] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240502 21:27:01.165445 7822 leader_election.cc:288] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:01.168567 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "845233761c2e45b0a23e25107f5461f1" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:01.168833 7349 raft_consensus.cc:3037] T 845233761c2e45b0a23e25107f5461f1 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.171994 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "845233761c2e45b0a23e25107f5461f1" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:01.172456 7349 raft_consensus.cc:2454] T 845233761c2e45b0a23e25107f5461f1 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:27:01.173156 7917 raft_consensus.cc:686] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:27:01.174960 7917 consensus_queue.cc:237] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:01.179143 4292 catalog_manager.cc:5484] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ddd486109f5434cace13d60d633ac04 (127.29.82.72). New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:01.179163 4300 catalog_manager.cc:5484] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:01.257587 7909 raft_consensus.cc:482] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:01.257799 7909 raft_consensus.cc:504] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:01.258595 7909 leader_election.cc:288] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:27:01.259069 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3528b68bc740ce9b452356fd26bfb6" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:01.259363 7618 raft_consensus.cc:2454] T 1d3528b68bc740ce9b452356fd26bfb6 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 0.
I20240502 21:27:01.261050 7282 leader_election.cc:302] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:01.261377 7909 raft_consensus.cc:2781] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:01.261500 7909 raft_consensus.cc:482] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:01.261637 7909 raft_consensus.cc:3037] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.264194 7909 raft_consensus.cc:504] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:01.264868 7909 leader_election.cc:288] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Requested vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:27:01.265213 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3528b68bc740ce9b452356fd26bfb6" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:01.265450 7618 raft_consensus.cc:3037] T 1d3528b68bc740ce9b452356fd26bfb6 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.266755 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3528b68bc740ce9b452356fd26bfb6" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:27:01.266824 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3528b68bc740ce9b452356fd26bfb6" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:27:01.267010 4574 raft_consensus.cc:2454] T 1d3528b68bc740ce9b452356fd26bfb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 0.
I20240502 21:27:01.267644 7618 raft_consensus.cc:2454] T 1d3528b68bc740ce9b452356fd26bfb6 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
I20240502 21:27:01.267989 7282 leader_election.cc:302] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, d0456c5e55a542128688904e003d403c; no voters: 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:27:01.268332 7909 raft_consensus.cc:2781] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:01.268705 7909 raft_consensus.cc:686] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Becoming Leader. State: Replica: d0456c5e55a542128688904e003d403c, State: Running, Role: LEADER
I20240502 21:27:01.269260 7909 consensus_queue.cc:237] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:01.272241 4300 catalog_manager.cc:5484] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c reported cstate change: term changed from 0 to 1, leader changed from <none> to d0456c5e55a542128688904e003d403c (127.29.82.70). New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:01.285811 7864 raft_consensus.cc:482] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:01.286026 7864 raft_consensus.cc:504] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:01.286883 7864 leader_election.cc:288] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:01.287384 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b1b8383bc64ae39b6ffe186be7b2c8" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:01.287698 7349 raft_consensus.cc:2454] T 74b1b8383bc64ae39b6ffe186be7b2c8 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:01.288297 7150 leader_election.cc:302] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:01.288944 7864 raft_consensus.cc:2781] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:01.289081 7864 raft_consensus.cc:482] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:01.289182 7864 raft_consensus.cc:3037] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.291209 7864 raft_consensus.cc:504] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:01.292016 7864 leader_election.cc:288] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:01.292275 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b1b8383bc64ae39b6ffe186be7b2c8" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:01.292529 7349 raft_consensus.cc:3037] T 74b1b8383bc64ae39b6ffe186be7b2c8 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:27:01.292548 4490 tablet.cc:2369] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 71e1375b799b439fa3c83ca763c039c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:27:01.295243 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b1b8383bc64ae39b6ffe186be7b2c8" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:27:01.295459 4443 raft_consensus.cc:3037] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.295593 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b1b8383bc64ae39b6ffe186be7b2c8" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:27:01.297314 7349 raft_consensus.cc:2454] T 74b1b8383bc64ae39b6ffe186be7b2c8 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:01.298398 4443 raft_consensus.cc:2454] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:01.298699 7150 leader_election.cc:302] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:01.299018 7864 raft_consensus.cc:2781] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:01.299479 7864 raft_consensus.cc:686] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:27:01.299839 7864 consensus_queue.cc:237] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:01.302649 4300 catalog_manager.cc:5484] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69). New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:01.427605 4300 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:01.435127 7349 raft_consensus.cc:1261] T 74b1b8383bc64ae39b6ffe186be7b2c8 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:01.435789 7864 consensus_queue.cc:1035] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:01.436082 4443 raft_consensus.cc:1261] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:01.436592 7864 consensus_queue.cc:1035] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:01.440011 4300 catalog_manager.cc:5576] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719) does not have the latest schema for tablet 74b1b8383bc64ae39b6ffe186be7b2c8 (table rebalancer_dynamic_table [id=59592ff43c694642872233217e7c82a9]). Expected version 1 got 0
I20240502 21:27:01.440554 7964 tablet.cc:1722] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:01.441411 7964 tablet.cc:1620] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:01.444180 4300 catalog_manager.cc:5576] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781) does not have the latest schema for tablet 74b1b8383bc64ae39b6ffe186be7b2c8 (table rebalancer_dynamic_table [id=59592ff43c694642872233217e7c82a9]). Expected version 1 got 0
I20240502 21:27:01.446733 4292 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240502 21:27:01.446964 4292 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240502 21:27:01.448943 4294 catalog_manager.cc:5576] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719) does not have the latest schema for tablet 74b1b8383bc64ae39b6ffe186be7b2c8 (table rebalancer_dynamic_table [id=59592ff43c694642872233217e7c82a9]). Expected version 1 got 0
I20240502 21:27:01.451601 4300 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=59592ff43c694642872233217e7c82a9] alter complete (version 1)
I20240502 21:27:01.451653 4292 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet e733125cd70d474cb75da730cc73c2ef
I20240502 21:27:01.452829 4422 tablet_service.cc:1514] Processing DeleteTablet for tablet e733125cd70d474cb75da730cc73c2ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:01 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:01.453136 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet e733125cd70d474cb75da730cc73c2ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:01 UTC) from {username='slave'} at 127.0.0.1:60018
I20240502 21:27:01.450063 7965 tablet.cc:1722] T 74b1b8383bc64ae39b6ffe186be7b2c8 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:01.453496 7717 tablet_replica.cc:330] T e733125cd70d474cb75da730cc73c2ef P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:27:01.453508 4817 tablet_service.cc:1514] Processing DeleteTablet for tablet e733125cd70d474cb75da730cc73c2ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:01 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:27:01.453771 7718 tablet_replica.cc:330] T e733125cd70d474cb75da730cc73c2ef P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:27:01.453804 7716 tablet_replica.cc:330] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:01.453823 7717 raft_consensus.cc:2229] T e733125cd70d474cb75da730cc73c2ef P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:01.453989 7716 raft_consensus.cc:2229] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:01.453982 7718 raft_consensus.cc:2229] T e733125cd70d474cb75da730cc73c2ef P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:01.454185 7717 raft_consensus.cc:2258] T e733125cd70d474cb75da730cc73c2ef P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:01.454247 7718 raft_consensus.cc:2258] T e733125cd70d474cb75da730cc73c2ef P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:01.454313 7716 raft_consensus.cc:2258] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:01.454962 7718 ts_tablet_manager.cc:1875] T e733125cd70d474cb75da730cc73c2ef P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:01.454957 7716 ts_tablet_manager.cc:1875] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:01.455127 7717 ts_tablet_manager.cc:1875] T e733125cd70d474cb75da730cc73c2ef P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:01.462854 7718 ts_tablet_manager.cc:1888] T e733125cd70d474cb75da730cc73c2ef P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:01.463042 7718 log.cc:1196] T e733125cd70d474cb75da730cc73c2ef P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/e733125cd70d474cb75da730cc73c2ef
I20240502 21:27:01.463593 7718 ts_tablet_manager.cc:1909] T e733125cd70d474cb75da730cc73c2ef P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:27:01.464802 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet e733125cd70d474cb75da730cc73c2ef (table rebalance_test_table_10 [id=b0aca53acbb943f78aa824def28f5058]) successfully deleted
I20240502 21:27:01.466961 7716 ts_tablet_manager.cc:1888] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:01.467213 4292 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet dfcd9d2ebaad42d2a2066a0f995f42a9
I20240502 21:27:01.467933 4292 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 1c49766c76da48b1b35c8837e5816a71
I20240502 21:27:01.468185 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet dfcd9d2ebaad42d2a2066a0f995f42a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:01 UTC) from {username='slave'} at 127.0.0.1:44764
I20240502 21:27:01.468436 4817 tablet_service.cc:1514] Processing DeleteTablet for tablet dfcd9d2ebaad42d2a2066a0f995f42a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:01 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:27:01.468477 7897 tablet_replica.cc:330] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:27:01.468677 7897 raft_consensus.cc:2229] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:01.468745 7718 tablet_replica.cc:330] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:27:01.468998 7897 raft_consensus.cc:2258] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:01.468971 7718 raft_consensus.cc:2229] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:01.469195 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c49766c76da48b1b35c8837e5816a71 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:01 UTC) from {username='slave'} at 127.0.0.1:60018
I20240502 21:27:01.469277 7718 raft_consensus.cc:2258] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:01.469738 7897 ts_tablet_manager.cc:1875] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:01.469758 7965 tablet.cc:1620] T 74b1b8383bc64ae39b6ffe186be7b2c8 P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:01.471493 7717 ts_tablet_manager.cc:1888] T e733125cd70d474cb75da730cc73c2ef P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:01.471630 7717 log.cc:1196] T e733125cd70d474cb75da730cc73c2ef P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/e733125cd70d474cb75da730cc73c2ef
I20240502 21:27:01.471763 7718 ts_tablet_manager.cc:1875] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:01.472121 7717 ts_tablet_manager.cc:1909] T e733125cd70d474cb75da730cc73c2ef P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting consensus metadata
I20240502 21:27:01.473398 4817 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c49766c76da48b1b35c8837e5816a71 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:01 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:27:01.473618 7717 tablet_replica.cc:330] T 1c49766c76da48b1b35c8837e5816a71 P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:27:01.473814 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet e733125cd70d474cb75da730cc73c2ef (table rebalance_test_table_10 [id=b0aca53acbb943f78aa824def28f5058]) successfully deleted
I20240502 21:27:01.473883 7717 raft_consensus.cc:2229] T 1c49766c76da48b1b35c8837e5816a71 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:01.468178 4422 tablet_service.cc:1514] Processing DeleteTablet for tablet dfcd9d2ebaad42d2a2066a0f995f42a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:01 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:01.474162 7717 raft_consensus.cc:2258] T 1c49766c76da48b1b35c8837e5816a71 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:01.471019 7966 tablet.cc:1722] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:01.472134 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c49766c76da48b1b35c8837e5816a71 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:01 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:01.474282 7716 log.cc:1196] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/e733125cd70d474cb75da730cc73c2ef
I20240502 21:27:01.474839 7716 ts_tablet_manager.cc:1909] T e733125cd70d474cb75da730cc73c2ef P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:01.475832 7966 tablet.cc:1620] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:01.475962 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet e733125cd70d474cb75da730cc73c2ef (table rebalance_test_table_10 [id=b0aca53acbb943f78aa824def28f5058]) successfully deleted
I20240502 21:27:01.476294 7897 ts_tablet_manager.cc:1888] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:01.476428 7897 log.cc:1196] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/dfcd9d2ebaad42d2a2066a0f995f42a9
I20240502 21:27:01.476871 7897 ts_tablet_manager.cc:1909] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 398c3c8f3195492c900439ad5e1931bb: Deleting consensus metadata
I20240502 21:27:01.479640 7718 ts_tablet_manager.cc:1888] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:01.479785 7718 log.cc:1196] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/dfcd9d2ebaad42d2a2066a0f995f42a9
I20240502 21:27:01.476264 7716 tablet_replica.cc:330] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:01.480062 7716 raft_consensus.cc:2229] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:01.480139 7718 ts_tablet_manager.cc:1909] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:27:01.480355 7716 raft_consensus.cc:2258] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:01.480901 7718 tablet_replica.cc:330] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:27:01.481020 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet dfcd9d2ebaad42d2a2066a0f995f42a9 (table rebalance_test_table_10 [id=b0aca53acbb943f78aa824def28f5058]) successfully deleted
I20240502 21:27:01.481122 7718 raft_consensus.cc:2229] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:01.481469 7718 raft_consensus.cc:2258] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:01.481536 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet dfcd9d2ebaad42d2a2066a0f995f42a9 (table rebalance_test_table_10 [id=b0aca53acbb943f78aa824def28f5058]) successfully deleted
I20240502 21:27:01.482131 7718 ts_tablet_manager.cc:1875] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:01.483659 7717 ts_tablet_manager.cc:1875] T 1c49766c76da48b1b35c8837e5816a71 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:01.486730 7718 ts_tablet_manager.cc:1888] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:01.486830 7718 log.cc:1196] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/1c49766c76da48b1b35c8837e5816a71
I20240502 21:27:01.487335 7718 ts_tablet_manager.cc:1909] T 1c49766c76da48b1b35c8837e5816a71 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:27:01.488481 7717 ts_tablet_manager.cc:1888] T 1c49766c76da48b1b35c8837e5816a71 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:01.488694 7717 log.cc:1196] T 1c49766c76da48b1b35c8837e5816a71 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/1c49766c76da48b1b35c8837e5816a71
I20240502 21:27:01.488689 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet 1c49766c76da48b1b35c8837e5816a71 (table rebalance_test_table_10 [id=b0aca53acbb943f78aa824def28f5058]) successfully deleted
I20240502 21:27:01.489099 7717 ts_tablet_manager.cc:1909] T 1c49766c76da48b1b35c8837e5816a71 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting consensus metadata
I20240502 21:27:01.489537 7716 ts_tablet_manager.cc:1875] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:01.490113 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet 1c49766c76da48b1b35c8837e5816a71 (table rebalance_test_table_10 [id=b0aca53acbb943f78aa824def28f5058]) successfully deleted
I20240502 21:27:01.491991 4294 catalog_manager.cc:5576] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781) does not have the latest schema for tablet 74b1b8383bc64ae39b6ffe186be7b2c8 (table rebalancer_dynamic_table [id=59592ff43c694642872233217e7c82a9]). Expected version 1 got 0
I20240502 21:27:01.496003 7716 ts_tablet_manager.cc:1888] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:01.496156 7716 log.cc:1196] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/dfcd9d2ebaad42d2a2066a0f995f42a9
I20240502 21:27:01.496671 7716 ts_tablet_manager.cc:1909] T dfcd9d2ebaad42d2a2066a0f995f42a9 P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:01.497565 7716 tablet_replica.cc:330] T 1c49766c76da48b1b35c8837e5816a71 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:01.497607 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet dfcd9d2ebaad42d2a2066a0f995f42a9 (table rebalance_test_table_10 [id=b0aca53acbb943f78aa824def28f5058]) successfully deleted
I20240502 21:27:01.497785 7716 raft_consensus.cc:2229] T 1c49766c76da48b1b35c8837e5816a71 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:01.498005 7716 raft_consensus.cc:2258] T 1c49766c76da48b1b35c8837e5816a71 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:01.498558 7716 ts_tablet_manager.cc:1875] T 1c49766c76da48b1b35c8837e5816a71 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:01.502480 7716 ts_tablet_manager.cc:1888] T 1c49766c76da48b1b35c8837e5816a71 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:01.502591 7716 log.cc:1196] T 1c49766c76da48b1b35c8837e5816a71 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/1c49766c76da48b1b35c8837e5816a71
I20240502 21:27:01.502948 7716 ts_tablet_manager.cc:1909] T 1c49766c76da48b1b35c8837e5816a71 P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:01.503790 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet 1c49766c76da48b1b35c8837e5816a71 (table rebalance_test_table_10 [id=b0aca53acbb943f78aa824def28f5058]) successfully deleted
I20240502 21:27:01.565189 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:01.566283 7917 consensus_queue.cc:1035] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:01.574301 7917 consensus_queue.cc:1035] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:01.575771 7964 tablet.cc:1722] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:01.576936 7965 tablet.cc:1722] T 74b1b8383bc64ae39b6ffe186be7b2c8 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:01.577435 7965 tablet.cc:1620] T 74b1b8383bc64ae39b6ffe186be7b2c8 P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:01.578612 7964 tablet.cc:1620] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:01.579746 7966 tablet.cc:1722] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:01.580287 7966 tablet.cc:1620] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:01.589322 4292 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=59592ff43c694642872233217e7c82a9] alter complete (version 2)
I20240502 21:27:01.609781 7755 consensus_queue.cc:1035] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:01.621063 7974 consensus_queue.cc:1035] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:01.686398 7909 consensus_queue.cc:1035] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:01.690349 4574 raft_consensus.cc:3037] T 1d3528b68bc740ce9b452356fd26bfb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.691455 4298 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:01.693691 7953 consensus_queue.cc:1035] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:01.700939 7964 tablet.cc:1722] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:01.701634 7965 tablet.cc:1722] T 74b1b8383bc64ae39b6ffe186be7b2c8 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:01.704205 7966 tablet.cc:1722] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:01.714164 4294 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=59592ff43c694642872233217e7c82a9] alter complete (version 3)
I20240502 21:27:01.788214 4294 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:01.797847 7330 tablet_service.cc:1467] Processing CreateTablet for tablet 9d279c5db74f46519c35fe75a35589c7 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=6a89f5df3f6a4844a9c775290e276fa0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:01.798223 4817 tablet_service.cc:1467] Processing CreateTablet for tablet ed822833ab3c451f96d722980b24c2b0 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=6a89f5df3f6a4844a9c775290e276fa0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:01.798355 7330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d279c5db74f46519c35fe75a35589c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.798662 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed822833ab3c451f96d722980b24c2b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.798664 7462 tablet_service.cc:1467] Processing CreateTablet for tablet 9d279c5db74f46519c35fe75a35589c7 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=6a89f5df3f6a4844a9c775290e276fa0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:01.799103 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d279c5db74f46519c35fe75a35589c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.803994 4555 tablet_service.cc:1467] Processing CreateTablet for tablet 9d279c5db74f46519c35fe75a35589c7 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=6a89f5df3f6a4844a9c775290e276fa0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:01.804425 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d279c5db74f46519c35fe75a35589c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.809348 7328 tablet_service.cc:1467] Processing CreateTablet for tablet ed822833ab3c451f96d722980b24c2b0 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=6a89f5df3f6a4844a9c775290e276fa0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:01.809780 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed822833ab3c451f96d722980b24c2b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.810024 7194 tablet_service.cc:1467] Processing CreateTablet for tablet ed822833ab3c451f96d722980b24c2b0 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=6a89f5df3f6a4844a9c775290e276fa0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:01.810593 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed822833ab3c451f96d722980b24c2b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.811378 4818 tablet_service.cc:1467] Processing CreateTablet for tablet 5bd324f8962647ef9e8fc01a64efe69a (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=6a89f5df3f6a4844a9c775290e276fa0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:01.811802 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bd324f8962647ef9e8fc01a64efe69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.815047 7195 tablet_service.cc:1467] Processing CreateTablet for tablet 5bd324f8962647ef9e8fc01a64efe69a (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=6a89f5df3f6a4844a9c775290e276fa0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:01.815441 7195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bd324f8962647ef9e8fc01a64efe69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.820679 7596 tablet_service.cc:1467] Processing CreateTablet for tablet 5bd324f8962647ef9e8fc01a64efe69a (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=6a89f5df3f6a4844a9c775290e276fa0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:01.821225 7596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bd324f8962647ef9e8fc01a64efe69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.822685 4294 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:01.822981 4294 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:01.832473 7982 tablet_bootstrap.cc:492] T 9d279c5db74f46519c35fe75a35589c7 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:01.834697 7982 tablet_bootstrap.cc:654] T 9d279c5db74f46519c35fe75a35589c7 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.840147 7980 tablet_bootstrap.cc:492] T 9d279c5db74f46519c35fe75a35589c7 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:01.841368 7985 tablet_bootstrap.cc:492] T ed822833ab3c451f96d722980b24c2b0 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:01.841584 7983 tablet_bootstrap.cc:492] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:01.841730 7982 tablet_bootstrap.cc:492] T 9d279c5db74f46519c35fe75a35589c7 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:01.841876 7982 ts_tablet_manager.cc:1367] T 9d279c5db74f46519c35fe75a35589c7 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240502 21:27:01.843343 7985 tablet_bootstrap.cc:654] T ed822833ab3c451f96d722980b24c2b0 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.844249 7984 tablet_bootstrap.cc:492] T 5bd324f8962647ef9e8fc01a64efe69a P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:01.844293 7980 tablet_bootstrap.cc:654] T 9d279c5db74f46519c35fe75a35589c7 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.844995 7983 tablet_bootstrap.cc:654] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.844590 4294 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 74b1b8383bc64ae39b6ffe186be7b2c8
I20240502 21:27:01.845471 7982 raft_consensus.cc:348] T 9d279c5db74f46519c35fe75a35589c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:01.845711 7982 raft_consensus.cc:374] T 9d279c5db74f46519c35fe75a35589c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.845805 7982 raft_consensus.cc:729] T 9d279c5db74f46519c35fe75a35589c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.846077 7982 consensus_queue.cc:260] T 9d279c5db74f46519c35fe75a35589c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:01.846346 7330 tablet_service.cc:1514] Processing DeleteTablet for tablet 74b1b8383bc64ae39b6ffe186be7b2c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:01 UTC) from {username='slave'} at 127.0.0.1:47078
I20240502 21:27:01.846575 7984 tablet_bootstrap.cc:654] T 5bd324f8962647ef9e8fc01a64efe69a P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.848101 7982 ts_tablet_manager.cc:1398] T 9d279c5db74f46519c35fe75a35589c7 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 21:27:01.848832 7704 raft_consensus.cc:482] T 9d279c5db74f46519c35fe75a35589c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:01.848954 7704 raft_consensus.cc:504] T 9d279c5db74f46519c35fe75a35589c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:01.849792 7704 leader_election.cc:288] T 9d279c5db74f46519c35fe75a35589c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:01.857601 7984 tablet_bootstrap.cc:492] T 5bd324f8962647ef9e8fc01a64efe69a P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:01.860625 7984 ts_tablet_manager.cc:1367] T 5bd324f8962647ef9e8fc01a64efe69a P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.003s
I20240502 21:27:01.861764 7984 raft_consensus.cc:348] T 5bd324f8962647ef9e8fc01a64efe69a P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:01.863737 7980 tablet_bootstrap.cc:492] T 9d279c5db74f46519c35fe75a35589c7 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:01.863955 7980 ts_tablet_manager.cc:1367] T 9d279c5db74f46519c35fe75a35589c7 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.003s
I20240502 21:27:01.863960 7983 tablet_bootstrap.cc:492] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:01.864127 7983 ts_tablet_manager.cc:1367] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.000s
I20240502 21:27:01.864373 7985 tablet_bootstrap.cc:492] T ed822833ab3c451f96d722980b24c2b0 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:01.864588 7985 ts_tablet_manager.cc:1367] T ed822833ab3c451f96d722980b24c2b0 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20240502 21:27:01.864923 7980 raft_consensus.cc:348] T 9d279c5db74f46519c35fe75a35589c7 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:01.865316 7980 raft_consensus.cc:374] T 9d279c5db74f46519c35fe75a35589c7 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.865487 7980 raft_consensus.cc:729] T 9d279c5db74f46519c35fe75a35589c7 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.865490 7985 raft_consensus.cc:348] T ed822833ab3c451f96d722980b24c2b0 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:01.865760 7985 raft_consensus.cc:374] T ed822833ab3c451f96d722980b24c2b0 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.865904 7985 raft_consensus.cc:729] T ed822833ab3c451f96d722980b24c2b0 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.865830 7980 consensus_queue.cc:260] T 9d279c5db74f46519c35fe75a35589c7 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:01.866212 7985 consensus_queue.cc:260] T ed822833ab3c451f96d722980b24c2b0 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:01.866250 7983 raft_consensus.cc:348] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:01.867153 7980 ts_tablet_manager.cc:1398] T 9d279c5db74f46519c35fe75a35589c7 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:01.867208 7983 raft_consensus.cc:374] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.867516 7980 tablet_bootstrap.cc:492] T ed822833ab3c451f96d722980b24c2b0 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:01.867528 7983 raft_consensus.cc:729] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.868130 7985 ts_tablet_manager.cc:1398] T ed822833ab3c451f96d722980b24c2b0 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:01.868018 7983 consensus_queue.cc:260] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:01.868463 7985 tablet_bootstrap.cc:492] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:01.868948 7988 tablet_replica.cc:330] T 74b1b8383bc64ae39b6ffe186be7b2c8 P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:27:01.869199 7984 raft_consensus.cc:374] T 5bd324f8962647ef9e8fc01a64efe69a P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.869558 7983 ts_tablet_manager.cc:1398] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:01.870401 7194 tablet_service.cc:1514] Processing DeleteTablet for tablet 74b1b8383bc64ae39b6ffe186be7b2c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:01 UTC) from {username='slave'} at 127.0.0.1:44652
I20240502 21:27:01.870128 7984 raft_consensus.cc:729] T 5bd324f8962647ef9e8fc01a64efe69a P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.871876 7984 consensus_queue.cc:260] T 5bd324f8962647ef9e8fc01a64efe69a P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:01.873303 7984 ts_tablet_manager.cc:1398] T 5bd324f8962647ef9e8fc01a64efe69a P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20240502 21:27:01.873467 7985 tablet_bootstrap.cc:654] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.875763 7985 tablet_bootstrap.cc:492] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:01.876004 7985 ts_tablet_manager.cc:1367] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20240502 21:27:01.876961 7985 raft_consensus.cc:348] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:01.877261 7985 raft_consensus.cc:374] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.877389 7985 raft_consensus.cc:729] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.877759 7985 consensus_queue.cc:260] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:01.879310 7985 ts_tablet_manager.cc:1398] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:01.880650 7195 tablet_service.cc:1514] Processing DeleteTablet for tablet 74b1b8383bc64ae39b6ffe186be7b2c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:01 UTC) from {username='slave'} at 127.0.0.1:44652
I20240502 21:27:01.880661 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet 74b1b8383bc64ae39b6ffe186be7b2c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:01 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:01.880967 7716 tablet_replica.cc:330] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:01.881206 7716 raft_consensus.cc:2229] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:01.881433 7716 raft_consensus.cc:2258] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:01.882637 7716 ts_tablet_manager.cc:1875] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:01.889277 7716 ts_tablet_manager.cc:1888] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:01.890484 7716 log.cc:1196] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/74b1b8383bc64ae39b6ffe186be7b2c8
I20240502 21:27:01.890877 7716 ts_tablet_manager.cc:1909] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:01.891919 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet 74b1b8383bc64ae39b6ffe186be7b2c8 (table rebalancer_dynamic_table_new_name [id=59592ff43c694642872233217e7c82a9]) successfully deleted
I20240502 21:27:01.894125 7983 tablet_bootstrap.cc:492] T 5bd324f8962647ef9e8fc01a64efe69a P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:01.886919 7947 raft_consensus.cc:482] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:01.894829 7947 raft_consensus.cc:504] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:01.895787 7996 tablet_replica.cc:330] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:27:01.896006 7996 raft_consensus.cc:2229] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:01.896018 7947 leader_election.cc:288] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:01.896648 7996 raft_consensus.cc:2258] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:01.897114 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed822833ab3c451f96d722980b24c2b0" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:01.897409 7215 raft_consensus.cc:2454] T ed822833ab3c451f96d722980b24c2b0 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:27:01.897909 4772 leader_election.cc:302] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:01.898305 7947 raft_consensus.cc:2781] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:01.898437 7947 raft_consensus.cc:482] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:01.898476 7983 tablet_bootstrap.cc:654] T 5bd324f8962647ef9e8fc01a64efe69a P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.898541 7947 raft_consensus.cc:3037] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.899843 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed822833ab3c451f96d722980b24c2b0" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:01.902530 7980 tablet_bootstrap.cc:654] T ed822833ab3c451f96d722980b24c2b0 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
W20240502 21:27:01.904413 4772 leader_election.cc:341] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:01.904771 7996 ts_tablet_manager.cc:1875] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:01.905273 7988 raft_consensus.cc:2229] T 74b1b8383bc64ae39b6ffe186be7b2c8 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:01.905727 7988 raft_consensus.cc:2258] T 74b1b8383bc64ae39b6ffe186be7b2c8 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:01.909932 7981 tablet_bootstrap.cc:492] T 9d279c5db74f46519c35fe75a35589c7 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:01.913631 7947 raft_consensus.cc:504] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:01.914816 7981 tablet_bootstrap.cc:654] T 9d279c5db74f46519c35fe75a35589c7 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.914963 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed822833ab3c451f96d722980b24c2b0" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
W20240502 21:27:01.915704 4772 leader_election.cc:341] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:01.915894 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d279c5db74f46519c35fe75a35589c7" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:01.916105 7350 raft_consensus.cc:2454] T 9d279c5db74f46519c35fe75a35589c7 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:27:01.916545 4510 leader_election.cc:302] T 9d279c5db74f46519c35fe75a35589c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a736a8ebb164ce0a051c0fccd76c3b8, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:01.917373 7704 raft_consensus.cc:2781] T 9d279c5db74f46519c35fe75a35589c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:01.917519 7704 raft_consensus.cc:482] T 9d279c5db74f46519c35fe75a35589c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:01.917615 7704 raft_consensus.cc:3037] T 9d279c5db74f46519c35fe75a35589c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.921145 7704 raft_consensus.cc:504] T 9d279c5db74f46519c35fe75a35589c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:01.921991 7704 leader_election.cc:288] T 9d279c5db74f46519c35fe75a35589c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:01.922366 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d279c5db74f46519c35fe75a35589c7" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:01.922560 7350 raft_consensus.cc:3037] T 9d279c5db74f46519c35fe75a35589c7 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.923707 7988 ts_tablet_manager.cc:1875] T 74b1b8383bc64ae39b6ffe186be7b2c8 P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:01.925401 7350 raft_consensus.cc:2454] T 9d279c5db74f46519c35fe75a35589c7 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:27:01.925853 4510 leader_election.cc:302] T 9d279c5db74f46519c35fe75a35589c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a736a8ebb164ce0a051c0fccd76c3b8, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:01.926157 7704 raft_consensus.cc:2781] T 9d279c5db74f46519c35fe75a35589c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:01.926456 7704 raft_consensus.cc:686] T 9d279c5db74f46519c35fe75a35589c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:27:01.926826 7704 consensus_queue.cc:237] T 9d279c5db74f46519c35fe75a35589c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:01.927768 7996 ts_tablet_manager.cc:1888] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:01.927928 7996 log.cc:1196] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/74b1b8383bc64ae39b6ffe186be7b2c8
I20240502 21:27:01.928479 7996 ts_tablet_manager.cc:1909] T 74b1b8383bc64ae39b6ffe186be7b2c8 P 39981a8dfe94462b9a73ba489af4c714: Deleting consensus metadata
I20240502 21:27:01.928921 7981 tablet_bootstrap.cc:492] T 9d279c5db74f46519c35fe75a35589c7 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:01.929085 7981 ts_tablet_manager.cc:1367] T 9d279c5db74f46519c35fe75a35589c7 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240502 21:27:01.929883 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet 74b1b8383bc64ae39b6ffe186be7b2c8 (table rebalancer_dynamic_table_new_name [id=59592ff43c694642872233217e7c82a9]) successfully deleted
I20240502 21:27:01.930625 7981 raft_consensus.cc:348] T 9d279c5db74f46519c35fe75a35589c7 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:01.930840 7981 raft_consensus.cc:374] T 9d279c5db74f46519c35fe75a35589c7 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.930938 7981 raft_consensus.cc:729] T 9d279c5db74f46519c35fe75a35589c7 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
W20240502 21:27:01.931115 4286 catalog_manager.cc:4797] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): delete failed for tablet 74b1b8383bc64ae39b6ffe186be7b2c8 because the tablet was not found. No further retry: Not found: Tablet not found: 74b1b8383bc64ae39b6ffe186be7b2c8
I20240502 21:27:01.931159 7947 leader_election.cc:288] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:01.931192 7981 consensus_queue.cc:260] T 9d279c5db74f46519c35fe75a35589c7 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:01.931831 7981 ts_tablet_manager.cc:1398] T 9d279c5db74f46519c35fe75a35589c7 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240502 21:27:01.931847 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed822833ab3c451f96d722980b24c2b0" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:01.932042 7215 raft_consensus.cc:3037] T ed822833ab3c451f96d722980b24c2b0 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.934648 7215 raft_consensus.cc:2454] T ed822833ab3c451f96d722980b24c2b0 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:27:01.935477 4772 leader_election.cc:302] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 73a1f2ca1f544de9b0e3b37015f85328; no voters: d0456c5e55a542128688904e003d403c
I20240502 21:27:01.936241 7947 raft_consensus.cc:2781] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:01.936517 7947 raft_consensus.cc:686] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:27:01.936888 7947 consensus_queue.cc:237] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:01.940344 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d279c5db74f46519c35fe75a35589c7" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:01.940587 7483 raft_consensus.cc:3037] T 9d279c5db74f46519c35fe75a35589c7 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.943758 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d279c5db74f46519c35fe75a35589c7" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:01.948350 7483 raft_consensus.cc:2454] T 9d279c5db74f46519c35fe75a35589c7 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:27:01.967283 7980 tablet_bootstrap.cc:492] T ed822833ab3c451f96d722980b24c2b0 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:01.967473 7980 ts_tablet_manager.cc:1367] T ed822833ab3c451f96d722980b24c2b0 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.100s user 0.010s sys 0.024s
I20240502 21:27:01.969630 4298 catalog_manager.cc:5484] T 9d279c5db74f46519c35fe75a35589c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:27:01.974979 7980 raft_consensus.cc:348] T ed822833ab3c451f96d722980b24c2b0 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:01.975320 7980 raft_consensus.cc:374] T ed822833ab3c451f96d722980b24c2b0 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.975471 7980 raft_consensus.cc:729] T ed822833ab3c451f96d722980b24c2b0 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.975836 7980 consensus_queue.cc:260] T ed822833ab3c451f96d722980b24c2b0 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:01.976537 7980 ts_tablet_manager.cc:1398] T ed822833ab3c451f96d722980b24c2b0 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 21:27:01.977694 4300 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240502 21:27:01.977905 4300 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240502 21:27:01.980618 7983 tablet_bootstrap.cc:492] T 5bd324f8962647ef9e8fc01a64efe69a P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:01.980852 7983 ts_tablet_manager.cc:1367] T 5bd324f8962647ef9e8fc01a64efe69a P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.087s user 0.014s sys 0.017s
I20240502 21:27:01.982322 4294 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:01.982959 7988 ts_tablet_manager.cc:1888] T 74b1b8383bc64ae39b6ffe186be7b2c8 P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:01.983124 7988 log.cc:1196] T 74b1b8383bc64ae39b6ffe186be7b2c8 P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/74b1b8383bc64ae39b6ffe186be7b2c8
I20240502 21:27:01.983671 7988 ts_tablet_manager.cc:1909] T 74b1b8383bc64ae39b6ffe186be7b2c8 P d0456c5e55a542128688904e003d403c: Deleting consensus metadata
I20240502 21:27:01.984413 7983 raft_consensus.cc:348] T 5bd324f8962647ef9e8fc01a64efe69a P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:01.985459 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet 74b1b8383bc64ae39b6ffe186be7b2c8 (table rebalancer_dynamic_table_new_name [id=59592ff43c694642872233217e7c82a9]) successfully deleted
I20240502 21:27:01.985703 7983 raft_consensus.cc:374] T 5bd324f8962647ef9e8fc01a64efe69a P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.985845 7983 raft_consensus.cc:729] T 5bd324f8962647ef9e8fc01a64efe69a P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.986264 7983 consensus_queue.cc:260] T 5bd324f8962647ef9e8fc01a64efe69a P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:01.986959 7983 ts_tablet_manager.cc:1398] T 5bd324f8962647ef9e8fc01a64efe69a P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 21:27:01.988613 4298 catalog_manager.cc:5484] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:01.990432 4300 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 61f757c73961428b863a29080a8f2652
I20240502 21:27:01.991240 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet 61f757c73961428b863a29080a8f2652 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:01 UTC) from {username='slave'} at 127.0.0.1:60018
I20240502 21:27:01.991715 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet 61f757c73961428b863a29080a8f2652 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:01 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:01.991966 8004 tablet_replica.cc:330] T 61f757c73961428b863a29080a8f2652 P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:27:01.992053 7716 tablet_replica.cc:330] T 61f757c73961428b863a29080a8f2652 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:01.992270 8004 raft_consensus.cc:2229] T 61f757c73961428b863a29080a8f2652 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:01.992278 7716 raft_consensus.cc:2229] T 61f757c73961428b863a29080a8f2652 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:01.992560 7716 raft_consensus.cc:2258] T 61f757c73961428b863a29080a8f2652 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:01.992595 8004 raft_consensus.cc:2258] T 61f757c73961428b863a29080a8f2652 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:01.993330 8004 ts_tablet_manager.cc:1875] T 61f757c73961428b863a29080a8f2652 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:01.993319 7716 ts_tablet_manager.cc:1875] T 61f757c73961428b863a29080a8f2652 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:01.997212 7194 tablet_service.cc:1467] Processing CreateTablet for tablet 2db7b755a3f741b28513abd6c81a2735 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e658963d7afc4ad6b7c6787855de9803]), partition=
I20240502 21:27:01.997588 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2db7b755a3f741b28513abd6c81a2735. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.999784 7716 ts_tablet_manager.cc:1888] T 61f757c73961428b863a29080a8f2652 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:01.999987 7716 log.cc:1196] T 61f757c73961428b863a29080a8f2652 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/61f757c73961428b863a29080a8f2652
I20240502 21:27:02.000492 7716 ts_tablet_manager.cc:1909] T 61f757c73961428b863a29080a8f2652 P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:02.001271 8004 ts_tablet_manager.cc:1888] T 61f757c73961428b863a29080a8f2652 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:02.001436 8004 log.cc:1196] T 61f757c73961428b863a29080a8f2652 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/61f757c73961428b863a29080a8f2652
I20240502 21:27:02.001914 8004 ts_tablet_manager.cc:1909] T 61f757c73961428b863a29080a8f2652 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting consensus metadata
I20240502 21:27:02.002986 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet 61f757c73961428b863a29080a8f2652 (table rebalance_test_table_12 [id=26f500760c604cc3aad27dd5cf163804]) successfully deleted
I20240502 21:27:02.003089 7985 tablet_bootstrap.cc:492] T 2db7b755a3f741b28513abd6c81a2735 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:02.003626 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet 61f757c73961428b863a29080a8f2652 (table rebalance_test_table_12 [id=26f500760c604cc3aad27dd5cf163804]) successfully deleted
I20240502 21:27:02.003773 7330 tablet_service.cc:1467] Processing CreateTablet for tablet 2db7b755a3f741b28513abd6c81a2735 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e658963d7afc4ad6b7c6787855de9803]), partition=
I20240502 21:27:02.004169 7330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2db7b755a3f741b28513abd6c81a2735. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.004518 7985 tablet_bootstrap.cc:654] T 2db7b755a3f741b28513abd6c81a2735 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:02.004567 4421 tablet_service.cc:1467] Processing CreateTablet for tablet 2db7b755a3f741b28513abd6c81a2735 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e658963d7afc4ad6b7c6787855de9803]), partition=
I20240502 21:27:02.004877 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2db7b755a3f741b28513abd6c81a2735. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.012009 4300 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet a2f37797d14b46c59834e9a888e018be
I20240502 21:27:02.012506 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet 61f757c73961428b863a29080a8f2652 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:01 UTC) from {username='slave'} at 127.0.0.1:44764
I20240502 21:27:02.012583 4300 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet b43650e973c745bda27b050ff5d4c5d7
I20240502 21:27:02.013726 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet a2f37797d14b46c59834e9a888e018be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:01 UTC) from {username='slave'} at 127.0.0.1:60018
I20240502 21:27:02.013733 4685 tablet_service.cc:1514] Processing DeleteTablet for tablet b43650e973c745bda27b050ff5d4c5d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:01 UTC) from {username='slave'} at 127.0.0.1:44764
I20240502 21:27:02.013985 8004 tablet_replica.cc:330] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:27:02.014302 8004 raft_consensus.cc:2229] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:02.014724 8004 raft_consensus.cc:2258] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:02.015277 4818 tablet_service.cc:1514] Processing DeleteTablet for tablet b43650e973c745bda27b050ff5d4c5d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:01 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:27:02.015972 4422 tablet_service.cc:1514] Processing DeleteTablet for tablet a2f37797d14b46c59834e9a888e018be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:01 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:02.016780 4817 tablet_service.cc:1514] Processing DeleteTablet for tablet a2f37797d14b46c59834e9a888e018be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:01 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:27:02.017127 4423 tablet_service.cc:1514] Processing DeleteTablet for tablet b43650e973c745bda27b050ff5d4c5d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:01 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:02.017330 8004 ts_tablet_manager.cc:1875] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:02.019007 7980 tablet_bootstrap.cc:492] T 2db7b755a3f741b28513abd6c81a2735 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:02.022693 7980 tablet_bootstrap.cc:654] T 2db7b755a3f741b28513abd6c81a2735 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:02.023492 7716 tablet_replica.cc:330] T a2f37797d14b46c59834e9a888e018be P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:02.024052 7716 raft_consensus.cc:2229] T a2f37797d14b46c59834e9a888e018be P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:02.025442 7716 raft_consensus.cc:2258] T a2f37797d14b46c59834e9a888e018be P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:02.027372 7716 ts_tablet_manager.cc:1875] T a2f37797d14b46c59834e9a888e018be P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:02.030352 8004 ts_tablet_manager.cc:1888] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:02.033285 8007 tablet_replica.cc:330] T b43650e973c745bda27b050ff5d4c5d7 P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:27:02.033576 8004 log.cc:1196] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/a2f37797d14b46c59834e9a888e018be
I20240502 21:27:02.034091 8004 ts_tablet_manager.cc:1909] T a2f37797d14b46c59834e9a888e018be P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting consensus metadata
I20240502 21:27:02.034902 8007 raft_consensus.cc:2229] T b43650e973c745bda27b050ff5d4c5d7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:02.035456 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet a2f37797d14b46c59834e9a888e018be (table rebalance_test_table_12 [id=26f500760c604cc3aad27dd5cf163804]) successfully deleted
I20240502 21:27:02.036204 8007 raft_consensus.cc:2258] T b43650e973c745bda27b050ff5d4c5d7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:02.036994 8007 ts_tablet_manager.cc:1875] T b43650e973c745bda27b050ff5d4c5d7 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:02.040694 7980 tablet_bootstrap.cc:492] T 2db7b755a3f741b28513abd6c81a2735 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:02.041321 7980 ts_tablet_manager.cc:1367] T 2db7b755a3f741b28513abd6c81a2735 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.003s
I20240502 21:27:02.045850 8006 tablet_replica.cc:330] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:27:02.046448 8006 raft_consensus.cc:2229] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:02.047923 7985 tablet_bootstrap.cc:492] T 2db7b755a3f741b28513abd6c81a2735 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:02.047834 7980 raft_consensus.cc:348] T 2db7b755a3f741b28513abd6c81a2735 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:02.048146 7985 ts_tablet_manager.cc:1367] T 2db7b755a3f741b28513abd6c81a2735 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.026s
I20240502 21:27:02.048637 7980 raft_consensus.cc:374] T 2db7b755a3f741b28513abd6c81a2735 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:02.048818 7980 raft_consensus.cc:729] T 2db7b755a3f741b28513abd6c81a2735 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:02.049017 7985 raft_consensus.cc:348] T 2db7b755a3f741b28513abd6c81a2735 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:02.049270 7985 raft_consensus.cc:374] T 2db7b755a3f741b28513abd6c81a2735 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:02.049188 7980 consensus_queue.cc:260] T 2db7b755a3f741b28513abd6c81a2735 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:02.048456 8006 raft_consensus.cc:2258] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:02.050475 7985 raft_consensus.cc:729] T 2db7b755a3f741b28513abd6c81a2735 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:02.050515 8006 ts_tablet_manager.cc:1875] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:02.050802 7985 consensus_queue.cc:260] T 2db7b755a3f741b28513abd6c81a2735 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:02.052309 7985 ts_tablet_manager.cc:1398] T 2db7b755a3f741b28513abd6c81a2735 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:02.052671 7980 ts_tablet_manager.cc:1398] T 2db7b755a3f741b28513abd6c81a2735 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240502 21:27:02.061196 8009 tablet_bootstrap.cc:492] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:02.062564 7716 ts_tablet_manager.cc:1888] T a2f37797d14b46c59834e9a888e018be P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:02.062739 7716 log.cc:1196] T a2f37797d14b46c59834e9a888e018be P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/a2f37797d14b46c59834e9a888e018be
I20240502 21:27:02.063030 8009 tablet_bootstrap.cc:654] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:02.063334 7716 ts_tablet_manager.cc:1909] T a2f37797d14b46c59834e9a888e018be P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:02.064132 7970 raft_consensus.cc:482] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:02.064291 7970 raft_consensus.cc:504] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:02.064872 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet a2f37797d14b46c59834e9a888e018be (table rebalance_test_table_12 [id=26f500760c604cc3aad27dd5cf163804]) successfully deleted
I20240502 21:27:02.065045 7970 leader_election.cc:288] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:02.065260 7716 tablet_replica.cc:330] T b43650e973c745bda27b050ff5d4c5d7 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:02.065470 7716 raft_consensus.cc:2229] T b43650e973c745bda27b050ff5d4c5d7 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:02.065730 7716 raft_consensus.cc:2258] T b43650e973c745bda27b050ff5d4c5d7 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:02.066226 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd324f8962647ef9e8fc01a64efe69a" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:27:02.066442 7716 ts_tablet_manager.cc:1875] T b43650e973c745bda27b050ff5d4c5d7 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:02.066468 4839 raft_consensus.cc:2454] T 5bd324f8962647ef9e8fc01a64efe69a P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:02.066875 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd324f8962647ef9e8fc01a64efe69a" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:02.067082 7618 raft_consensus.cc:2454] T 5bd324f8962647ef9e8fc01a64efe69a P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:02.067972 7148 leader_election.cc:302] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:02.068111 8007 ts_tablet_manager.cc:1888] T b43650e973c745bda27b050ff5d4c5d7 P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:02.068233 8007 log.cc:1196] T b43650e973c745bda27b050ff5d4c5d7 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/b43650e973c745bda27b050ff5d4c5d7
I20240502 21:27:02.068251 7970 raft_consensus.cc:2781] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:02.068360 7970 raft_consensus.cc:482] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:02.068446 7970 raft_consensus.cc:3037] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:02.069890 8007 ts_tablet_manager.cc:1909] T b43650e973c745bda27b050ff5d4c5d7 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:27:02.070830 8007 tablet_replica.cc:330] T a2f37797d14b46c59834e9a888e018be P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:27:02.071044 8007 raft_consensus.cc:2229] T a2f37797d14b46c59834e9a888e018be P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:02.071110 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet b43650e973c745bda27b050ff5d4c5d7 (table rebalance_test_table_12 [id=26f500760c604cc3aad27dd5cf163804]) successfully deleted
I20240502 21:27:02.071930 8006 ts_tablet_manager.cc:1888] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:02.072213 8007 raft_consensus.cc:2258] T a2f37797d14b46c59834e9a888e018be P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:02.072206 7970 raft_consensus.cc:504] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:02.072394 8006 log.cc:1196] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/61f757c73961428b863a29080a8f2652
I20240502 21:27:02.072656 7716 ts_tablet_manager.cc:1888] T b43650e973c745bda27b050ff5d4c5d7 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:02.072808 7716 log.cc:1196] T b43650e973c745bda27b050ff5d4c5d7 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/b43650e973c745bda27b050ff5d4c5d7
I20240502 21:27:02.072870 8006 ts_tablet_manager.cc:1909] T 61f757c73961428b863a29080a8f2652 P 398c3c8f3195492c900439ad5e1931bb: Deleting consensus metadata
I20240502 21:27:02.073293 7716 ts_tablet_manager.cc:1909] T b43650e973c745bda27b050ff5d4c5d7 P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:02.073779 8007 ts_tablet_manager.cc:1875] T a2f37797d14b46c59834e9a888e018be P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:02.074481 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet b43650e973c745bda27b050ff5d4c5d7 (table rebalance_test_table_12 [id=26f500760c604cc3aad27dd5cf163804]) successfully deleted
I20240502 21:27:02.074574 7970 leader_election.cc:288] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:02.073695 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd324f8962647ef9e8fc01a64efe69a" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:27:02.074690 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd324f8962647ef9e8fc01a64efe69a" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:02.074762 4839 raft_consensus.cc:3037] T 5bd324f8962647ef9e8fc01a64efe69a P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:02.074915 7618 raft_consensus.cc:3037] T 5bd324f8962647ef9e8fc01a64efe69a P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:02.076763 8006 tablet_replica.cc:330] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:27:02.077075 8006 raft_consensus.cc:2229] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:02.077073 4839 raft_consensus.cc:2454] T 5bd324f8962647ef9e8fc01a64efe69a P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:02.077700 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet 61f757c73961428b863a29080a8f2652 (table rebalance_test_table_12 [id=26f500760c604cc3aad27dd5cf163804]) successfully deleted
I20240502 21:27:02.077901 8006 raft_consensus.cc:2258] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:02.078022 7618 raft_consensus.cc:2454] T 5bd324f8962647ef9e8fc01a64efe69a P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:02.078015 7148 leader_election.cc:302] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:02.078433 7970 raft_consensus.cc:2781] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:02.078590 8009 tablet_bootstrap.cc:492] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:02.078576 7970 raft_consensus.cc:686] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:27:02.078724 8009 ts_tablet_manager.cc:1367] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.009s
I20240502 21:27:02.078713 8006 ts_tablet_manager.cc:1875] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:02.078863 7970 consensus_queue.cc:237] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:02.079293 8009 raft_consensus.cc:348] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:02.079470 8009 raft_consensus.cc:374] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:02.079537 8009 raft_consensus.cc:729] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:02.079795 8009 consensus_queue.cc:260] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:02.080413 8009 ts_tablet_manager.cc:1398] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:02.083377 4298 catalog_manager.cc:5484] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69). New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:02.086798 8007 ts_tablet_manager.cc:1888] T a2f37797d14b46c59834e9a888e018be P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:02.087003 8007 log.cc:1196] T a2f37797d14b46c59834e9a888e018be P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/a2f37797d14b46c59834e9a888e018be
I20240502 21:27:02.087471 8007 ts_tablet_manager.cc:1909] T a2f37797d14b46c59834e9a888e018be P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:27:02.087926 8006 ts_tablet_manager.cc:1888] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:02.088129 8006 log.cc:1196] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/b43650e973c745bda27b050ff5d4c5d7
I20240502 21:27:02.088551 8006 ts_tablet_manager.cc:1909] T b43650e973c745bda27b050ff5d4c5d7 P 398c3c8f3195492c900439ad5e1931bb: Deleting consensus metadata
I20240502 21:27:02.088624 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet a2f37797d14b46c59834e9a888e018be (table rebalance_test_table_12 [id=26f500760c604cc3aad27dd5cf163804]) successfully deleted
I20240502 21:27:02.089471 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet b43650e973c745bda27b050ff5d4c5d7 (table rebalance_test_table_12 [id=26f500760c604cc3aad27dd5cf163804]) successfully deleted
I20240502 21:27:02.100155 7953 raft_consensus.cc:482] T ed822833ab3c451f96d722980b24c2b0 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:02.100355 7953 raft_consensus.cc:504] T ed822833ab3c451f96d722980b24c2b0 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:02.101099 7953 leader_election.cc:288] T ed822833ab3c451f96d722980b24c2b0 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:02.101836 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed822833ab3c451f96d722980b24c2b0" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:02.102093 7215 raft_consensus.cc:2379] T ed822833ab3c451f96d722980b24c2b0 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0456c5e55a542128688904e003d403c in current term 1: Already voted for candidate 73a1f2ca1f544de9b0e3b37015f85328 in this term.
I20240502 21:27:02.113210 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed822833ab3c451f96d722980b24c2b0" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:27:02.113807 7281 leader_election.cc:302] T ed822833ab3c451f96d722980b24c2b0 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0456c5e55a542128688904e003d403c; no voters: 39981a8dfe94462b9a73ba489af4c714, 73a1f2ca1f544de9b0e3b37015f85328
I20240502 21:27:02.114179 7953 raft_consensus.cc:3037] T ed822833ab3c451f96d722980b24c2b0 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:02.116798 7953 raft_consensus.cc:2726] T ed822833ab3c451f96d722980b24c2b0 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:27:02.242740 7939 raft_consensus.cc:482] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:02.242928 7939 raft_consensus.cc:504] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:02.243710 7939 leader_election.cc:288] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:02.250571 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db7b755a3f741b28513abd6c81a2735" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:02.250752 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db7b755a3f741b28513abd6c81a2735" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:02.250831 7350 raft_consensus.cc:2454] T 2db7b755a3f741b28513abd6c81a2735 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:02.250993 7215 raft_consensus.cc:2454] T 2db7b755a3f741b28513abd6c81a2735 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:02.251333 4377 leader_election.cc:302] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:02.251634 7939 raft_consensus.cc:2781] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:02.251762 7939 raft_consensus.cc:482] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:02.251860 7939 raft_consensus.cc:3037] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:02.253754 7939 raft_consensus.cc:504] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:02.254271 7939 leader_election.cc:288] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:02.254652 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db7b755a3f741b28513abd6c81a2735" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:02.254730 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db7b755a3f741b28513abd6c81a2735" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:02.254809 7350 raft_consensus.cc:3037] T 2db7b755a3f741b28513abd6c81a2735 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:02.254923 7215 raft_consensus.cc:3037] T 2db7b755a3f741b28513abd6c81a2735 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:02.257349 7350 raft_consensus.cc:2454] T 2db7b755a3f741b28513abd6c81a2735 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:02.257524 7215 raft_consensus.cc:2454] T 2db7b755a3f741b28513abd6c81a2735 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:02.258019 4377 leader_election.cc:302] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:02.258355 7939 raft_consensus.cc:2781] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:02.258914 7939 raft_consensus.cc:686] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:27:02.259229 7939 consensus_queue.cc:237] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:02.262799 4298 catalog_manager.cc:5484] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:02.321244 7917 consensus_queue.cc:1035] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:02.327054 7917 consensus_queue.cc:1035] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:02.415920 4298 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:02.422477 7350 raft_consensus.cc:1261] T 2db7b755a3f741b28513abd6c81a2735 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:02.423194 7939 consensus_queue.cc:1035] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:02.423758 7215 raft_consensus.cc:1261] T 2db7b755a3f741b28513abd6c81a2735 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:02.424763 7939 consensus_queue.cc:1035] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:02.428438 4300 catalog_manager.cc:5576] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719) does not have the latest schema for tablet 2db7b755a3f741b28513abd6c81a2735 (table rebalancer_dynamic_table [id=e658963d7afc4ad6b7c6787855de9803]). Expected version 1 got 0
I20240502 21:27:02.430289 4300 catalog_manager.cc:5576] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193) does not have the latest schema for tablet 2db7b755a3f741b28513abd6c81a2735 (table rebalancer_dynamic_table [id=e658963d7afc4ad6b7c6787855de9803]). Expected version 1 got 0
I20240502 21:27:02.433938 8031 tablet.cc:1722] T 2db7b755a3f741b28513abd6c81a2735 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:02.436776 4298 catalog_manager.cc:5576] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193) does not have the latest schema for tablet 2db7b755a3f741b28513abd6c81a2735 (table rebalancer_dynamic_table [id=e658963d7afc4ad6b7c6787855de9803]). Expected version 1 got 0
I20240502 21:27:02.438784 8030 tablet.cc:1722] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:02.439105 8032 tablet.cc:1722] T 2db7b755a3f741b28513abd6c81a2735 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:02.439486 8030 tablet.cc:1620] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:02.439819 8032 tablet.cc:1620] T 2db7b755a3f741b28513abd6c81a2735 P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:02.440109 8031 tablet.cc:1620] T 2db7b755a3f741b28513abd6c81a2735 P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:02.440587 4298 catalog_manager.cc:5576] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719) does not have the latest schema for tablet 2db7b755a3f741b28513abd6c81a2735 (table rebalancer_dynamic_table [id=e658963d7afc4ad6b7c6787855de9803]). Expected version 1 got 0
I20240502 21:27:02.447008 4298 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=e658963d7afc4ad6b7c6787855de9803] alter complete (version 1)
I20240502 21:27:02.495065 8010 consensus_queue.cc:1035] T 9d279c5db74f46519c35fe75a35589c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:02.503459 8010 consensus_queue.cc:1035] T 9d279c5db74f46519c35fe75a35589c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:02.515704 4298 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240502 21:27:02.515945 4298 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240502 21:27:02.519018 4298 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 7e5dd46893db452c88b5c250153674da
I20240502 21:27:02.519481 4298 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 35da5719247c4901a8521f8cb063a7b1
I20240502 21:27:02.519899 4818 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e5dd46893db452c88b5c250153674da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:02 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:27:02.519927 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e5dd46893db452c88b5c250153674da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:02 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:02.519963 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e5dd46893db452c88b5c250153674da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:02 UTC) from {username='slave'} at 127.0.0.1:60018
I20240502 21:27:02.520241 7716 tablet_replica.cc:330] T 7e5dd46893db452c88b5c250153674da P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:02.520242 8007 tablet_replica.cc:330] T 7e5dd46893db452c88b5c250153674da P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:27:02.520252 4298 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 69816765eeb84db1871d2873d40215cd
I20240502 21:27:02.520483 8007 raft_consensus.cc:2229] T 7e5dd46893db452c88b5c250153674da P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:02.520496 7716 raft_consensus.cc:2229] T 7e5dd46893db452c88b5c250153674da P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:02.520737 8007 raft_consensus.cc:2258] T 7e5dd46893db452c88b5c250153674da P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:02.520889 7716 raft_consensus.cc:2258] T 7e5dd46893db452c88b5c250153674da P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:02.521265 4817 tablet_service.cc:1514] Processing DeleteTablet for tablet 69816765eeb84db1871d2873d40215cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:02 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:27:02.521628 8007 ts_tablet_manager.cc:1875] T 7e5dd46893db452c88b5c250153674da P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:02.521628 7716 ts_tablet_manager.cc:1875] T 7e5dd46893db452c88b5c250153674da P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:02.524011 8004 tablet_replica.cc:330] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:27:02.523991 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet 35da5719247c4901a8521f8cb063a7b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:02 UTC) from {username='slave'} at 127.0.0.1:44764
I20240502 21:27:02.524241 8004 raft_consensus.cc:2229] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:02.524329 8006 tablet_replica.cc:330] T 35da5719247c4901a8521f8cb063a7b1 P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:27:02.524348 4818 tablet_service.cc:1514] Processing DeleteTablet for tablet 35da5719247c4901a8521f8cb063a7b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:02 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:27:02.524539 8006 raft_consensus.cc:2229] T 35da5719247c4901a8521f8cb063a7b1 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:02.524657 8004 raft_consensus.cc:2258] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:02.524845 8006 raft_consensus.cc:2258] T 35da5719247c4901a8521f8cb063a7b1 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:02.525020 4423 tablet_service.cc:1514] Processing DeleteTablet for tablet 69816765eeb84db1871d2873d40215cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:02 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:02.525431 8004 ts_tablet_manager.cc:1875] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:02.525591 8006 ts_tablet_manager.cc:1875] T 35da5719247c4901a8521f8cb063a7b1 P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:02.528337 7716 ts_tablet_manager.cc:1888] T 7e5dd46893db452c88b5c250153674da P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:02.528489 7716 log.cc:1196] T 7e5dd46893db452c88b5c250153674da P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/7e5dd46893db452c88b5c250153674da
I20240502 21:27:02.528491 8007 ts_tablet_manager.cc:1888] T 7e5dd46893db452c88b5c250153674da P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:02.528604 8007 log.cc:1196] T 7e5dd46893db452c88b5c250153674da P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/7e5dd46893db452c88b5c250153674da
I20240502 21:27:02.528875 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet 69816765eeb84db1871d2873d40215cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:02 UTC) from {username='slave'} at 127.0.0.1:60018
I20240502 21:27:02.529093 7716 ts_tablet_manager.cc:1909] T 7e5dd46893db452c88b5c250153674da P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:02.529255 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet 35da5719247c4901a8521f8cb063a7b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:02 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:02.529325 8007 ts_tablet_manager.cc:1909] T 7e5dd46893db452c88b5c250153674da P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:27:02.530253 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet 7e5dd46893db452c88b5c250153674da (table rebalance_test_table_14 [id=54c217fb12554f2689ff67970d9771e9]) successfully deleted
I20240502 21:27:02.530405 8007 tablet_replica.cc:330] T 69816765eeb84db1871d2873d40215cd P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:27:02.530611 7716 tablet_replica.cc:330] T 69816765eeb84db1871d2873d40215cd P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:02.530611 8007 raft_consensus.cc:2229] T 69816765eeb84db1871d2873d40215cd P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:02.530802 8007 raft_consensus.cc:2258] T 69816765eeb84db1871d2873d40215cd P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:02.530790 7716 raft_consensus.cc:2229] T 69816765eeb84db1871d2873d40215cd P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:02.531013 7716 raft_consensus.cc:2258] T 69816765eeb84db1871d2873d40215cd P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:02.531467 8007 ts_tablet_manager.cc:1875] T 69816765eeb84db1871d2873d40215cd P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:02.531666 7716 ts_tablet_manager.cc:1875] T 69816765eeb84db1871d2873d40215cd P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:02.532522 8006 ts_tablet_manager.cc:1888] T 35da5719247c4901a8521f8cb063a7b1 P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:02.532632 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet 7e5dd46893db452c88b5c250153674da (table rebalance_test_table_14 [id=54c217fb12554f2689ff67970d9771e9]) successfully deleted
I20240502 21:27:02.532685 8006 log.cc:1196] T 35da5719247c4901a8521f8cb063a7b1 P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/35da5719247c4901a8521f8cb063a7b1
I20240502 21:27:02.533166 8006 ts_tablet_manager.cc:1909] T 35da5719247c4901a8521f8cb063a7b1 P 398c3c8f3195492c900439ad5e1931bb: Deleting consensus metadata
I20240502 21:27:02.534209 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet 35da5719247c4901a8521f8cb063a7b1 (table rebalance_test_table_14 [id=54c217fb12554f2689ff67970d9771e9]) successfully deleted
I20240502 21:27:02.538166 8004 ts_tablet_manager.cc:1888] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:02.538211 7716 ts_tablet_manager.cc:1888] T 69816765eeb84db1871d2873d40215cd P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:02.538285 8004 log.cc:1196] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/7e5dd46893db452c88b5c250153674da
I20240502 21:27:02.538312 7716 log.cc:1196] T 69816765eeb84db1871d2873d40215cd P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/69816765eeb84db1871d2873d40215cd
I20240502 21:27:02.538748 8004 ts_tablet_manager.cc:1909] T 7e5dd46893db452c88b5c250153674da P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting consensus metadata
I20240502 21:27:02.538825 7716 ts_tablet_manager.cc:1909] T 69816765eeb84db1871d2873d40215cd P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:02.538825 8007 ts_tablet_manager.cc:1888] T 69816765eeb84db1871d2873d40215cd P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:02.538957 8007 log.cc:1196] T 69816765eeb84db1871d2873d40215cd P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/69816765eeb84db1871d2873d40215cd
I20240502 21:27:02.539602 8007 ts_tablet_manager.cc:1909] T 69816765eeb84db1871d2873d40215cd P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:27:02.539777 7716 tablet_replica.cc:330] T 35da5719247c4901a8521f8cb063a7b1 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:02.539988 7716 raft_consensus.cc:2229] T 35da5719247c4901a8521f8cb063a7b1 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:02.540086 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet 69816765eeb84db1871d2873d40215cd (table rebalance_test_table_14 [id=54c217fb12554f2689ff67970d9771e9]) successfully deleted
I20240502 21:27:02.540199 7716 raft_consensus.cc:2258] T 35da5719247c4901a8521f8cb063a7b1 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:02.540561 8004 tablet_replica.cc:330] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:27:02.540751 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet 7e5dd46893db452c88b5c250153674da (table rebalance_test_table_14 [id=54c217fb12554f2689ff67970d9771e9]) successfully deleted
I20240502 21:27:02.540793 8004 raft_consensus.cc:2229] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:02.540880 7716 ts_tablet_manager.cc:1875] T 35da5719247c4901a8521f8cb063a7b1 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:02.541116 8004 raft_consensus.cc:2258] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:02.541826 8004 ts_tablet_manager.cc:1875] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:02.542843 8007 tablet_replica.cc:330] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:27:02.542888 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet 69816765eeb84db1871d2873d40215cd (table rebalance_test_table_14 [id=54c217fb12554f2689ff67970d9771e9]) successfully deleted
I20240502 21:27:02.543045 8007 raft_consensus.cc:2229] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:02.543377 8007 raft_consensus.cc:2258] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:02.544073 8007 ts_tablet_manager.cc:1875] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:02.546648 7716 ts_tablet_manager.cc:1888] T 35da5719247c4901a8521f8cb063a7b1 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:02.546782 7716 log.cc:1196] T 35da5719247c4901a8521f8cb063a7b1 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/35da5719247c4901a8521f8cb063a7b1
I20240502 21:27:02.547284 7716 ts_tablet_manager.cc:1909] T 35da5719247c4901a8521f8cb063a7b1 P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:02.548310 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet 35da5719247c4901a8521f8cb063a7b1 (table rebalance_test_table_14 [id=54c217fb12554f2689ff67970d9771e9]) successfully deleted
I20240502 21:27:02.547969 8004 ts_tablet_manager.cc:1888] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:02.548753 8004 log.cc:1196] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/69816765eeb84db1871d2873d40215cd
I20240502 21:27:02.549135 8004 ts_tablet_manager.cc:1909] T 69816765eeb84db1871d2873d40215cd P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting consensus metadata
I20240502 21:27:02.549932 8007 ts_tablet_manager.cc:1888] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:02.549937 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet 69816765eeb84db1871d2873d40215cd (table rebalance_test_table_14 [id=54c217fb12554f2689ff67970d9771e9]) successfully deleted
I20240502 21:27:02.550046 8007 log.cc:1196] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/35da5719247c4901a8521f8cb063a7b1
I20240502 21:27:02.550408 8007 ts_tablet_manager.cc:1909] T 35da5719247c4901a8521f8cb063a7b1 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:27:02.551153 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet 35da5719247c4901a8521f8cb063a7b1 (table rebalance_test_table_14 [id=54c217fb12554f2689ff67970d9771e9]) successfully deleted
I20240502 21:27:02.553221 4298 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:02.560356 8030 tablet.cc:1722] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:02.560880 8030 tablet.cc:1620] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:02.561298 8031 tablet.cc:1722] T 2db7b755a3f741b28513abd6c81a2735 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:02.561759 8031 tablet.cc:1620] T 2db7b755a3f741b28513abd6c81a2735 P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:02.562029 8032 tablet.cc:1722] T 2db7b755a3f741b28513abd6c81a2735 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:02.562491 8032 tablet.cc:1620] T 2db7b755a3f741b28513abd6c81a2735 P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:02.570367 4298 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=e658963d7afc4ad6b7c6787855de9803] alter complete (version 2)
I20240502 21:27:02.595790 4298 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:02.604950 7462 tablet_service.cc:1467] Processing CreateTablet for tablet 648263320b474a32bfffa18b1d4a61e3 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=a99691f2b4cb42afb50b56d30958d085]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:02.605141 7330 tablet_service.cc:1467] Processing CreateTablet for tablet 648263320b474a32bfffa18b1d4a61e3 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=a99691f2b4cb42afb50b56d30958d085]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:02.605487 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 648263320b474a32bfffa18b1d4a61e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.605594 7330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 648263320b474a32bfffa18b1d4a61e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.606743 4555 tablet_service.cc:1467] Processing CreateTablet for tablet 171a71f8620c44909b78cb7b35cfed5e (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=a99691f2b4cb42afb50b56d30958d085]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:02.607187 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 171a71f8620c44909b78cb7b35cfed5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.608126 7329 tablet_service.cc:1467] Processing CreateTablet for tablet 0f822150741f4ec1a3a9472572160359 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=a99691f2b4cb42afb50b56d30958d085]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:02.608510 7329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f822150741f4ec1a3a9472572160359. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.611018 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 0f822150741f4ec1a3a9472572160359 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=a99691f2b4cb42afb50b56d30958d085]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:02.611116 7328 tablet_service.cc:1467] Processing CreateTablet for tablet 171a71f8620c44909b78cb7b35cfed5e (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=a99691f2b4cb42afb50b56d30958d085]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:02.611481 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f822150741f4ec1a3a9472572160359. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.611476 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 171a71f8620c44909b78cb7b35cfed5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.612183 4421 tablet_service.cc:1467] Processing CreateTablet for tablet 171a71f8620c44909b78cb7b35cfed5e (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=a99691f2b4cb42afb50b56d30958d085]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:02.612658 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 171a71f8620c44909b78cb7b35cfed5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.615995 4818 tablet_service.cc:1467] Processing CreateTablet for tablet 648263320b474a32bfffa18b1d4a61e3 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=a99691f2b4cb42afb50b56d30958d085]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:02.616021 7464 tablet_service.cc:1467] Processing CreateTablet for tablet 0f822150741f4ec1a3a9472572160359 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=a99691f2b4cb42afb50b56d30958d085]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:02.616422 7464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f822150741f4ec1a3a9472572160359. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.616427 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 648263320b474a32bfffa18b1d4a61e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.622092 8037 tablet_bootstrap.cc:492] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:02.623919 8037 tablet_bootstrap.cc:654] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:02.631747 8043 tablet_bootstrap.cc:492] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:02.634301 8040 tablet_bootstrap.cc:492] T 171a71f8620c44909b78cb7b35cfed5e P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:02.636587 8040 tablet_bootstrap.cc:654] T 171a71f8620c44909b78cb7b35cfed5e P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:02.640206 8042 tablet_bootstrap.cc:492] T 171a71f8620c44909b78cb7b35cfed5e P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:02.640498 8043 tablet_bootstrap.cc:654] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:02.640826 8044 tablet_bootstrap.cc:492] T 171a71f8620c44909b78cb7b35cfed5e P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:02.641572 8041 tablet_bootstrap.cc:492] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:02.643038 8044 tablet_bootstrap.cc:654] T 171a71f8620c44909b78cb7b35cfed5e P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:02.644317 8041 tablet_bootstrap.cc:654] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:02.644954 8042 tablet_bootstrap.cc:654] T 171a71f8620c44909b78cb7b35cfed5e P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:02.647697 8043 tablet_bootstrap.cc:492] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:27:02.648331 8043 ts_tablet_manager.cc:1367] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.001s
I20240502 21:27:02.648432 8040 tablet_bootstrap.cc:492] T 171a71f8620c44909b78cb7b35cfed5e P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:02.648636 8040 ts_tablet_manager.cc:1367] T 171a71f8620c44909b78cb7b35cfed5e P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20240502 21:27:02.649628 8040 raft_consensus.cc:348] T 171a71f8620c44909b78cb7b35cfed5e P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:02.649912 8040 raft_consensus.cc:374] T 171a71f8620c44909b78cb7b35cfed5e P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:02.649677 8043 raft_consensus.cc:348] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:02.650019 8040 raft_consensus.cc:729] T 171a71f8620c44909b78cb7b35cfed5e P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:02.650203 8043 raft_consensus.cc:374] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:02.650393 8043 raft_consensus.cc:729] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:02.650290 8040 consensus_queue.cc:260] T 171a71f8620c44909b78cb7b35cfed5e P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:02.651130 8040 ts_tablet_manager.cc:1398] T 171a71f8620c44909b78cb7b35cfed5e P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:02.653674 8041 tablet_bootstrap.cc:492] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:02.653887 8041 ts_tablet_manager.cc:1367] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.003s
I20240502 21:27:02.653709 8043 consensus_queue.cc:260] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:02.654637 8043 ts_tablet_manager.cc:1398] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240502 21:27:02.654804 8041 raft_consensus.cc:348] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:02.655211 8041 raft_consensus.cc:374] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:02.655376 8041 raft_consensus.cc:729] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:02.655407 7970 consensus_queue.cc:1035] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:02.655752 8041 consensus_queue.cc:260] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:02.656469 8041 ts_tablet_manager.cc:1398] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:02.661217 7970 consensus_queue.cc:1035] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:02.661893 8042 tablet_bootstrap.cc:492] T 171a71f8620c44909b78cb7b35cfed5e P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:02.662175 8042 ts_tablet_manager.cc:1367] T 171a71f8620c44909b78cb7b35cfed5e P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.000s
I20240502 21:27:02.663270 8042 raft_consensus.cc:348] T 171a71f8620c44909b78cb7b35cfed5e P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:02.663655 8042 raft_consensus.cc:374] T 171a71f8620c44909b78cb7b35cfed5e P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:02.664008 8042 raft_consensus.cc:729] T 171a71f8620c44909b78cb7b35cfed5e P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:02.664413 8042 consensus_queue.cc:260] T 171a71f8620c44909b78cb7b35cfed5e P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:02.665751 8042 ts_tablet_manager.cc:1398] T 171a71f8620c44909b78cb7b35cfed5e P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:02.665789 8037 tablet_bootstrap.cc:492] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:02.666127 8042 tablet_bootstrap.cc:492] T 0f822150741f4ec1a3a9472572160359 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:02.667853 8037 ts_tablet_manager.cc:1367] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.046s user 0.015s sys 0.005s
I20240502 21:27:02.668385 8044 tablet_bootstrap.cc:492] T 171a71f8620c44909b78cb7b35cfed5e P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:02.668545 8044 ts_tablet_manager.cc:1367] T 171a71f8620c44909b78cb7b35cfed5e P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.000s
I20240502 21:27:02.669028 8042 tablet_bootstrap.cc:654] T 0f822150741f4ec1a3a9472572160359 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:02.669469 8044 raft_consensus.cc:348] T 171a71f8620c44909b78cb7b35cfed5e P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:02.669829 8044 raft_consensus.cc:374] T 171a71f8620c44909b78cb7b35cfed5e P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:02.669939 8044 raft_consensus.cc:729] T 171a71f8620c44909b78cb7b35cfed5e P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:02.670188 8044 consensus_queue.cc:260] T 171a71f8620c44909b78cb7b35cfed5e P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:02.669822 8037 raft_consensus.cc:348] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:02.671943 8037 raft_consensus.cc:374] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:02.672087 8037 raft_consensus.cc:729] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:02.672416 8037 consensus_queue.cc:260] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:02.673738 8037 ts_tablet_manager.cc:1398] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20240502 21:27:02.674052 8037 tablet_bootstrap.cc:492] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:02.675855 8037 tablet_bootstrap.cc:654] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:02.676039 8044 ts_tablet_manager.cc:1398] T 171a71f8620c44909b78cb7b35cfed5e P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240502 21:27:02.679510 4300 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:02.684101 8037 tablet_bootstrap.cc:492] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:02.684396 8037 ts_tablet_manager.cc:1367] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20240502 21:27:02.685448 8037 raft_consensus.cc:348] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:02.685784 8037 raft_consensus.cc:374] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:02.685914 8037 raft_consensus.cc:729] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:02.686365 8042 tablet_bootstrap.cc:492] T 0f822150741f4ec1a3a9472572160359 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:02.686293 8037 consensus_queue.cc:260] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:02.686547 8042 ts_tablet_manager.cc:1367] T 0f822150741f4ec1a3a9472572160359 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.006s
I20240502 21:27:02.687606 8037 ts_tablet_manager.cc:1398] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:02.688916 8042 raft_consensus.cc:348] T 0f822150741f4ec1a3a9472572160359 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:02.689175 8042 raft_consensus.cc:374] T 0f822150741f4ec1a3a9472572160359 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:02.689260 8042 raft_consensus.cc:729] T 0f822150741f4ec1a3a9472572160359 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:02.690816 8031 tablet.cc:1722] T 2db7b755a3f741b28513abd6c81a2735 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:02.690925 8030 tablet.cc:1722] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:02.692512 8042 consensus_queue.cc:260] T 0f822150741f4ec1a3a9472572160359 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:02.693797 8042 ts_tablet_manager.cc:1398] T 0f822150741f4ec1a3a9472572160359 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 21:27:02.694116 8042 tablet_bootstrap.cc:492] T 648263320b474a32bfffa18b1d4a61e3 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:02.696223 8032 tablet.cc:1722] T 2db7b755a3f741b28513abd6c81a2735 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:02.696336 8042 tablet_bootstrap.cc:654] T 648263320b474a32bfffa18b1d4a61e3 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:02.701196 8024 raft_consensus.cc:482] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:02.701380 8024 raft_consensus.cc:504] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:02.702229 8024 leader_election.cc:288] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:02.702598 4298 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=e658963d7afc4ad6b7c6787855de9803] alter complete (version 3)
I20240502 21:27:02.702662 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "648263320b474a32bfffa18b1d4a61e3" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:02.702912 7484 raft_consensus.cc:2454] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:27:02.703529 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "648263320b474a32bfffa18b1d4a61e3" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
W20240502 21:27:02.704649 4772 leader_election.cc:341] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:02.704993 4770 leader_election.cc:302] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 73a1f2ca1f544de9b0e3b37015f85328; no voters: d0456c5e55a542128688904e003d403c
I20240502 21:27:02.705363 8024 raft_consensus.cc:2781] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:02.705518 8024 raft_consensus.cc:482] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:02.705632 8024 raft_consensus.cc:3037] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:02.709383 8024 raft_consensus.cc:504] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:02.710306 8024 leader_election.cc:288] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:02.710798 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "648263320b474a32bfffa18b1d4a61e3" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:02.710822 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "648263320b474a32bfffa18b1d4a61e3" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:02.711030 7484 raft_consensus.cc:3037] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:02.713788 7484 raft_consensus.cc:2454] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
W20240502 21:27:02.713948 4772 leader_election.cc:341] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:02.714529 4770 leader_election.cc:302] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 73a1f2ca1f544de9b0e3b37015f85328; no voters: d0456c5e55a542128688904e003d403c
I20240502 21:27:02.714985 8024 raft_consensus.cc:2781] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:02.715205 8024 raft_consensus.cc:686] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:27:02.715543 8024 consensus_queue.cc:237] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:02.716159 8042 tablet_bootstrap.cc:492] T 648263320b474a32bfffa18b1d4a61e3 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:02.716315 8042 ts_tablet_manager.cc:1367] T 648263320b474a32bfffa18b1d4a61e3 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.000s
I20240502 21:27:02.717072 8042 raft_consensus.cc:348] T 648263320b474a32bfffa18b1d4a61e3 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:02.717308 8042 raft_consensus.cc:374] T 648263320b474a32bfffa18b1d4a61e3 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:02.717415 8042 raft_consensus.cc:729] T 648263320b474a32bfffa18b1d4a61e3 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:02.717691 8042 consensus_queue.cc:260] T 648263320b474a32bfffa18b1d4a61e3 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:02.718295 8042 ts_tablet_manager.cc:1398] T 648263320b474a32bfffa18b1d4a61e3 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:02.721707 4300 catalog_manager.cc:5484] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:02.766314 8010 raft_consensus.cc:482] T 171a71f8620c44909b78cb7b35cfed5e P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:02.766479 8010 raft_consensus.cc:504] T 171a71f8620c44909b78cb7b35cfed5e P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:02.767294 8010 leader_election.cc:288] T 171a71f8620c44909b78cb7b35cfed5e P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:02.767800 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171a71f8620c44909b78cb7b35cfed5e" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:02.768045 7350 raft_consensus.cc:2454] T 171a71f8620c44909b78cb7b35cfed5e P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:27:02.768087 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171a71f8620c44909b78cb7b35cfed5e" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:27:02.768316 4443 raft_consensus.cc:2454] T 171a71f8620c44909b78cb7b35cfed5e P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:27:02.768728 4508 leader_election.cc:302] T 171a71f8620c44909b78cb7b35cfed5e P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:27:02.769026 8010 raft_consensus.cc:2781] T 171a71f8620c44909b78cb7b35cfed5e P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:02.769132 8010 raft_consensus.cc:482] T 171a71f8620c44909b78cb7b35cfed5e P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:02.769222 8010 raft_consensus.cc:3037] T 171a71f8620c44909b78cb7b35cfed5e P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:02.771215 8010 raft_consensus.cc:504] T 171a71f8620c44909b78cb7b35cfed5e P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:02.771991 8010 leader_election.cc:288] T 171a71f8620c44909b78cb7b35cfed5e P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:02.772323 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171a71f8620c44909b78cb7b35cfed5e" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:02.772523 7350 raft_consensus.cc:3037] T 171a71f8620c44909b78cb7b35cfed5e P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:02.772519 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171a71f8620c44909b78cb7b35cfed5e" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:27:02.772760 4443 raft_consensus.cc:3037] T 171a71f8620c44909b78cb7b35cfed5e P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:02.774710 4443 raft_consensus.cc:2454] T 171a71f8620c44909b78cb7b35cfed5e P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:27:02.775153 4508 leader_election.cc:302] T 171a71f8620c44909b78cb7b35cfed5e P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:27:02.775367 7350 raft_consensus.cc:2454] T 171a71f8620c44909b78cb7b35cfed5e P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:27:02.775461 8010 raft_consensus.cc:2781] T 171a71f8620c44909b78cb7b35cfed5e P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:02.775647 8010 raft_consensus.cc:686] T 171a71f8620c44909b78cb7b35cfed5e P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:27:02.776203 8010 consensus_queue.cc:237] T 171a71f8620c44909b78cb7b35cfed5e P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:02.779294 4294 catalog_manager.cc:5484] T 171a71f8620c44909b78cb7b35cfed5e P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:02.807565 4294 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:02.807833 4294 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:02.811619 4294 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 2db7b755a3f741b28513abd6c81a2735
I20240502 21:27:02.812489 7328 tablet_service.cc:1514] Processing DeleteTablet for tablet 2db7b755a3f741b28513abd6c81a2735 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:02 UTC) from {username='slave'} at 127.0.0.1:47078
I20240502 21:27:02.812826 7194 tablet_service.cc:1514] Processing DeleteTablet for tablet 2db7b755a3f741b28513abd6c81a2735 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:02 UTC) from {username='slave'} at 127.0.0.1:44652
I20240502 21:27:02.815404 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet 2db7b755a3f741b28513abd6c81a2735 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:02 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:02.815749 7716 tablet_replica.cc:330] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:02.815974 7716 raft_consensus.cc:2229] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:02.816315 7716 raft_consensus.cc:2258] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:02.817512 7716 ts_tablet_manager.cc:1875] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:02.823916 8060 tablet_replica.cc:330] T 2db7b755a3f741b28513abd6c81a2735 P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:27:02.824143 8060 raft_consensus.cc:2229] T 2db7b755a3f741b28513abd6c81a2735 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:02.824376 8060 raft_consensus.cc:2258] T 2db7b755a3f741b28513abd6c81a2735 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:02.826550 8059 tablet_replica.cc:330] T 2db7b755a3f741b28513abd6c81a2735 P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:27:02.826692 7716 ts_tablet_manager.cc:1888] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:02.826889 7716 log.cc:1196] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/2db7b755a3f741b28513abd6c81a2735
I20240502 21:27:02.826993 8059 raft_consensus.cc:2229] T 2db7b755a3f741b28513abd6c81a2735 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:02.827279 8059 raft_consensus.cc:2258] T 2db7b755a3f741b28513abd6c81a2735 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:02.827472 7716 ts_tablet_manager.cc:1909] T 2db7b755a3f741b28513abd6c81a2735 P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:02.828037 8060 ts_tablet_manager.cc:1875] T 2db7b755a3f741b28513abd6c81a2735 P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:02.828227 8059 ts_tablet_manager.cc:1875] T 2db7b755a3f741b28513abd6c81a2735 P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:02.829128 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet 2db7b755a3f741b28513abd6c81a2735 (table rebalancer_dynamic_table_new_name [id=e658963d7afc4ad6b7c6787855de9803]) successfully deleted
I20240502 21:27:02.833447 8060 ts_tablet_manager.cc:1888] T 2db7b755a3f741b28513abd6c81a2735 P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:02.833565 8060 log.cc:1196] T 2db7b755a3f741b28513abd6c81a2735 P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/2db7b755a3f741b28513abd6c81a2735
I20240502 21:27:02.834008 8060 ts_tablet_manager.cc:1909] T 2db7b755a3f741b28513abd6c81a2735 P 39981a8dfe94462b9a73ba489af4c714: Deleting consensus metadata
I20240502 21:27:02.834370 8059 ts_tablet_manager.cc:1888] T 2db7b755a3f741b28513abd6c81a2735 P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:02.834548 8059 log.cc:1196] T 2db7b755a3f741b28513abd6c81a2735 P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/2db7b755a3f741b28513abd6c81a2735
I20240502 21:27:02.835048 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet 2db7b755a3f741b28513abd6c81a2735 (table rebalancer_dynamic_table_new_name [id=e658963d7afc4ad6b7c6787855de9803]) successfully deleted
I20240502 21:27:02.835134 8059 ts_tablet_manager.cc:1909] T 2db7b755a3f741b28513abd6c81a2735 P d0456c5e55a542128688904e003d403c: Deleting consensus metadata
I20240502 21:27:02.836637 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet 2db7b755a3f741b28513abd6c81a2735 (table rebalancer_dynamic_table_new_name [id=e658963d7afc4ad6b7c6787855de9803]) successfully deleted
I20240502 21:27:02.886626 7534 raft_consensus.cc:482] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:02.886843 7534 raft_consensus.cc:504] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:02.887766 7534 leader_election.cc:288] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:02.908053 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f822150741f4ec1a3a9472572160359" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:02.908327 7350 raft_consensus.cc:2454] T 0f822150741f4ec1a3a9472572160359 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:27:02.908807 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f822150741f4ec1a3a9472572160359" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:02.909046 7484 raft_consensus.cc:2454] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:27:02.909433 4641 leader_election.cc:302] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:02.909754 7534 raft_consensus.cc:2781] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:02.909895 7534 raft_consensus.cc:482] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:02.909996 7534 raft_consensus.cc:3037] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:02.912355 7534 raft_consensus.cc:504] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:02.912938 7534 leader_election.cc:288] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Requested vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:02.913360 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f822150741f4ec1a3a9472572160359" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:02.913586 7350 raft_consensus.cc:3037] T 0f822150741f4ec1a3a9472572160359 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:02.913615 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f822150741f4ec1a3a9472572160359" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:02.913842 7484 raft_consensus.cc:3037] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:02.916671 7484 raft_consensus.cc:2454] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:27:02.916869 7350 raft_consensus.cc:2454] T 0f822150741f4ec1a3a9472572160359 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:27:02.916767 4294 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:02.917310 4639 leader_election.cc:302] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 398c3c8f3195492c900439ad5e1931bb; no voters:
I20240502 21:27:02.917668 7534 raft_consensus.cc:2781] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:02.917852 7534 raft_consensus.cc:686] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Becoming Leader. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Running, Role: LEADER
I20240502 21:27:02.918159 7534 consensus_queue.cc:237] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:02.923890 4300 catalog_manager.cc:5484] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 398c3c8f3195492c900439ad5e1931bb (127.29.82.67). New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:02.925212 7194 tablet_service.cc:1467] Processing CreateTablet for tablet 4c5712eac5c549c3bad16072ec699edc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=031b7edd28b641bcae41e11bcb6fcb9a]), partition=
I20240502 21:27:02.925380 7328 tablet_service.cc:1467] Processing CreateTablet for tablet 4c5712eac5c549c3bad16072ec699edc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=031b7edd28b641bcae41e11bcb6fcb9a]), partition=
I20240502 21:27:02.925725 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c5712eac5c549c3bad16072ec699edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.925750 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c5712eac5c549c3bad16072ec699edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.926688 7464 tablet_service.cc:1467] Processing CreateTablet for tablet 4c5712eac5c549c3bad16072ec699edc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=031b7edd28b641bcae41e11bcb6fcb9a]), partition=
I20240502 21:27:02.927047 7464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c5712eac5c549c3bad16072ec699edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.932605 8042 tablet_bootstrap.cc:492] T 4c5712eac5c549c3bad16072ec699edc P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:02.933948 8042 tablet_bootstrap.cc:654] T 4c5712eac5c549c3bad16072ec699edc P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:02.936061 8042 tablet_bootstrap.cc:492] T 4c5712eac5c549c3bad16072ec699edc P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:02.936228 8042 ts_tablet_manager.cc:1367] T 4c5712eac5c549c3bad16072ec699edc P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:02.936805 8067 tablet_bootstrap.cc:492] T 4c5712eac5c549c3bad16072ec699edc P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:02.936983 8042 raft_consensus.cc:348] T 4c5712eac5c549c3bad16072ec699edc P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:02.937240 8042 raft_consensus.cc:374] T 4c5712eac5c549c3bad16072ec699edc P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:02.937415 8042 raft_consensus.cc:729] T 4c5712eac5c549c3bad16072ec699edc P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:02.937783 8042 consensus_queue.cc:260] T 4c5712eac5c549c3bad16072ec699edc P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:02.937986 8037 tablet_bootstrap.cc:492] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:02.938485 8042 ts_tablet_manager.cc:1398] T 4c5712eac5c549c3bad16072ec699edc P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:02.939478 8037 tablet_bootstrap.cc:654] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:02.940655 8067 tablet_bootstrap.cc:654] T 4c5712eac5c549c3bad16072ec699edc P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:02.943185 8067 tablet_bootstrap.cc:492] T 4c5712eac5c549c3bad16072ec699edc P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:02.943377 8067 ts_tablet_manager.cc:1367] T 4c5712eac5c549c3bad16072ec699edc P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 21:27:02.944265 8067 raft_consensus.cc:348] T 4c5712eac5c549c3bad16072ec699edc P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:02.944542 8067 raft_consensus.cc:374] T 4c5712eac5c549c3bad16072ec699edc P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:02.944648 8067 raft_consensus.cc:729] T 4c5712eac5c549c3bad16072ec699edc P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:02.944931 8067 consensus_queue.cc:260] T 4c5712eac5c549c3bad16072ec699edc P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:02.945449 8067 ts_tablet_manager.cc:1398] T 4c5712eac5c549c3bad16072ec699edc P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:02.950012 8037 tablet_bootstrap.cc:492] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:02.950150 8037 ts_tablet_manager.cc:1367] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.006s
I20240502 21:27:02.950742 8037 raft_consensus.cc:348] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:02.950942 8037 raft_consensus.cc:374] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:02.951036 8037 raft_consensus.cc:729] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:02.951243 8037 consensus_queue.cc:260] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:02.951803 8037 ts_tablet_manager.cc:1398] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:02.993745 7832 raft_consensus.cc:482] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:02.993940 7832 raft_consensus.cc:504] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:02.994648 7832 leader_election.cc:288] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:02.995976 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c5712eac5c549c3bad16072ec699edc" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:02.996268 7214 raft_consensus.cc:2454] T 4c5712eac5c549c3bad16072ec699edc P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:02.996793 7419 leader_election.cc:302] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:02.997134 7832 raft_consensus.cc:2781] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:02.997256 7832 raft_consensus.cc:482] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:02.997364 7832 raft_consensus.cc:3037] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:02.999723 7832 raft_consensus.cc:504] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:03.001099 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c5712eac5c549c3bad16072ec699edc" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:03.001358 7214 raft_consensus.cc:3037] T 4c5712eac5c549c3bad16072ec699edc P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.004221 7214 raft_consensus.cc:2454] T 4c5712eac5c549c3bad16072ec699edc P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:03.004439 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c5712eac5c549c3bad16072ec699edc" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:03.004727 7349 raft_consensus.cc:3037] T 4c5712eac5c549c3bad16072ec699edc P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.006724 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c5712eac5c549c3bad16072ec699edc" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:03.007349 7349 raft_consensus.cc:2454] T 4c5712eac5c549c3bad16072ec699edc P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:03.007781 7832 leader_election.cc:288] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:03.008211 7419 leader_election.cc:302] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:03.008497 7832 raft_consensus.cc:2781] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:03.008733 7832 raft_consensus.cc:686] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Becoming Leader. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Running, Role: LEADER
I20240502 21:27:03.009080 7832 consensus_queue.cc:237] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:03.012774 4300 catalog_manager.cc:5484] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a72286397c4edc9a63eca3412be168 (127.29.82.71). New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:27:03.025115 4300 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240502 21:27:03.025406 4300 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240502 21:27:03.029292 4300 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet a59e4a0405bc4cac81c9e95bd8d14662
I20240502 21:27:03.030134 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet a59e4a0405bc4cac81c9e95bd8d14662 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:03 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:03.030256 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet a59e4a0405bc4cac81c9e95bd8d14662 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:03 UTC) from {username='slave'} at 127.0.0.1:60018
I20240502 21:27:03.030440 7716 tablet_replica.cc:330] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:03.030674 7716 raft_consensus.cc:2229] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:03.030718 8004 tablet_replica.cc:330] T a59e4a0405bc4cac81c9e95bd8d14662 P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:27:03.030916 8004 raft_consensus.cc:2229] T a59e4a0405bc4cac81c9e95bd8d14662 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:03.031026 7716 raft_consensus.cc:2258] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:03.031142 8004 raft_consensus.cc:2258] T a59e4a0405bc4cac81c9e95bd8d14662 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:03.031129 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet a59e4a0405bc4cac81c9e95bd8d14662 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:03 UTC) from {username='slave'} at 127.0.0.1:44764
I20240502 21:27:03.031415 8006 tablet_replica.cc:330] T a59e4a0405bc4cac81c9e95bd8d14662 P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:27:03.031668 8006 raft_consensus.cc:2229] T a59e4a0405bc4cac81c9e95bd8d14662 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:03.031857 7716 ts_tablet_manager.cc:1875] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:03.031889 8004 ts_tablet_manager.cc:1875] T a59e4a0405bc4cac81c9e95bd8d14662 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:03.031949 8006 raft_consensus.cc:2258] T a59e4a0405bc4cac81c9e95bd8d14662 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:03.032682 8006 ts_tablet_manager.cc:1875] T a59e4a0405bc4cac81c9e95bd8d14662 P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:03.038501 4300 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 8b5d021c9ba34930b2efb654e47ab20d
I20240502 21:27:03.039173 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b5d021c9ba34930b2efb654e47ab20d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:03 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:03.039516 7716 ts_tablet_manager.cc:1888] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:03.039615 4300 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 445f931ca1364bb193759d9c6e4ebac9
I20240502 21:27:03.039748 4818 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b5d021c9ba34930b2efb654e47ab20d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:03 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:27:03.039808 7716 log.cc:1196] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/a59e4a0405bc4cac81c9e95bd8d14662
I20240502 21:27:03.040033 8007 tablet_replica.cc:330] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:27:03.040246 7716 ts_tablet_manager.cc:1909] T a59e4a0405bc4cac81c9e95bd8d14662 P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:03.040264 8007 raft_consensus.cc:2229] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:03.040275 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet 445f931ca1364bb193759d9c6e4ebac9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:03 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:03.040629 8007 raft_consensus.cc:2258] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:03.040683 4818 tablet_service.cc:1514] Processing DeleteTablet for tablet 445f931ca1364bb193759d9c6e4ebac9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:03 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:27:03.039889 8004 ts_tablet_manager.cc:1888] T a59e4a0405bc4cac81c9e95bd8d14662 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:03.039559 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b5d021c9ba34930b2efb654e47ab20d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:03 UTC) from {username='slave'} at 127.0.0.1:60018
I20240502 21:27:03.041266 8004 log.cc:1196] T a59e4a0405bc4cac81c9e95bd8d14662 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/a59e4a0405bc4cac81c9e95bd8d14662
I20240502 21:27:03.041366 8007 ts_tablet_manager.cc:1875] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:03.041446 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet a59e4a0405bc4cac81c9e95bd8d14662 (table rebalance_test_table_16 [id=78a4b23f64324443a6125591bda63185]) successfully deleted
I20240502 21:27:03.041721 8004 ts_tablet_manager.cc:1909] T a59e4a0405bc4cac81c9e95bd8d14662 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting consensus metadata
I20240502 21:27:03.041785 7716 tablet_replica.cc:330] T 8b5d021c9ba34930b2efb654e47ab20d P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:03.041967 7716 raft_consensus.cc:2229] T 8b5d021c9ba34930b2efb654e47ab20d P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:03.041996 8006 ts_tablet_manager.cc:1888] T a59e4a0405bc4cac81c9e95bd8d14662 P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:03.042131 8006 log.cc:1196] T a59e4a0405bc4cac81c9e95bd8d14662 P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/a59e4a0405bc4cac81c9e95bd8d14662
I20240502 21:27:03.042165 7716 raft_consensus.cc:2258] T 8b5d021c9ba34930b2efb654e47ab20d P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:03.042528 8006 ts_tablet_manager.cc:1909] T a59e4a0405bc4cac81c9e95bd8d14662 P 398c3c8f3195492c900439ad5e1931bb: Deleting consensus metadata
I20240502 21:27:03.042708 8004 tablet_replica.cc:330] T 8b5d021c9ba34930b2efb654e47ab20d P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:27:03.042876 7716 ts_tablet_manager.cc:1875] T 8b5d021c9ba34930b2efb654e47ab20d P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:03.042912 8004 raft_consensus.cc:2229] T 8b5d021c9ba34930b2efb654e47ab20d P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:03.043128 8004 raft_consensus.cc:2258] T 8b5d021c9ba34930b2efb654e47ab20d P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:03.043534 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet a59e4a0405bc4cac81c9e95bd8d14662 (table rebalance_test_table_16 [id=78a4b23f64324443a6125591bda63185]) successfully deleted
I20240502 21:27:03.043828 8004 ts_tablet_manager.cc:1875] T 8b5d021c9ba34930b2efb654e47ab20d P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:03.043846 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet 445f931ca1364bb193759d9c6e4ebac9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:03 UTC) from {username='slave'} at 127.0.0.1:44764
I20240502 21:27:03.044301 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet a59e4a0405bc4cac81c9e95bd8d14662 (table rebalance_test_table_16 [id=78a4b23f64324443a6125591bda63185]) successfully deleted
I20240502 21:27:03.044607 8006 tablet_replica.cc:330] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:27:03.044865 8006 raft_consensus.cc:2229] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:03.045236 8006 raft_consensus.cc:2258] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:03.045975 8006 ts_tablet_manager.cc:1875] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:03.050457 7716 ts_tablet_manager.cc:1888] T 8b5d021c9ba34930b2efb654e47ab20d P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:03.050619 7716 log.cc:1196] T 8b5d021c9ba34930b2efb654e47ab20d P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/8b5d021c9ba34930b2efb654e47ab20d
I20240502 21:27:03.051086 7716 ts_tablet_manager.cc:1909] T 8b5d021c9ba34930b2efb654e47ab20d P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:03.051321 8007 ts_tablet_manager.cc:1888] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:03.051527 8007 log.cc:1196] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/8b5d021c9ba34930b2efb654e47ab20d
I20240502 21:27:03.052029 8007 ts_tablet_manager.cc:1909] T 8b5d021c9ba34930b2efb654e47ab20d P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:27:03.052346 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet 8b5d021c9ba34930b2efb654e47ab20d (table rebalance_test_table_16 [id=78a4b23f64324443a6125591bda63185]) successfully deleted
I20240502 21:27:03.052423 8004 ts_tablet_manager.cc:1888] T 8b5d021c9ba34930b2efb654e47ab20d P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:03.052595 8004 log.cc:1196] T 8b5d021c9ba34930b2efb654e47ab20d P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/8b5d021c9ba34930b2efb654e47ab20d
I20240502 21:27:03.052785 7716 tablet_replica.cc:330] T 445f931ca1364bb193759d9c6e4ebac9 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:03.052867 8006 ts_tablet_manager.cc:1888] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:03.053028 8006 log.cc:1196] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/445f931ca1364bb193759d9c6e4ebac9
I20240502 21:27:03.053069 8007 tablet_replica.cc:330] T 445f931ca1364bb193759d9c6e4ebac9 P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:27:03.053110 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet 8b5d021c9ba34930b2efb654e47ab20d (table rebalance_test_table_16 [id=78a4b23f64324443a6125591bda63185]) successfully deleted
I20240502 21:27:03.053344 8007 raft_consensus.cc:2229] T 445f931ca1364bb193759d9c6e4ebac9 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:03.053376 7716 raft_consensus.cc:2229] T 445f931ca1364bb193759d9c6e4ebac9 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:03.053498 8006 ts_tablet_manager.cc:1909] T 445f931ca1364bb193759d9c6e4ebac9 P 398c3c8f3195492c900439ad5e1931bb: Deleting consensus metadata
I20240502 21:27:03.053622 7716 raft_consensus.cc:2258] T 445f931ca1364bb193759d9c6e4ebac9 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:03.053654 8007 raft_consensus.cc:2258] T 445f931ca1364bb193759d9c6e4ebac9 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:03.053417 8004 ts_tablet_manager.cc:1909] T 8b5d021c9ba34930b2efb654e47ab20d P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting consensus metadata
I20240502 21:27:03.054334 7716 ts_tablet_manager.cc:1875] T 445f931ca1364bb193759d9c6e4ebac9 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:03.054343 8007 ts_tablet_manager.cc:1875] T 445f931ca1364bb193759d9c6e4ebac9 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:03.054546 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet 445f931ca1364bb193759d9c6e4ebac9 (table rebalance_test_table_16 [id=78a4b23f64324443a6125591bda63185]) successfully deleted
I20240502 21:27:03.054929 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet 8b5d021c9ba34930b2efb654e47ab20d (table rebalance_test_table_16 [id=78a4b23f64324443a6125591bda63185]) successfully deleted
I20240502 21:27:03.060819 7716 ts_tablet_manager.cc:1888] T 445f931ca1364bb193759d9c6e4ebac9 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:03.060811 8007 ts_tablet_manager.cc:1888] T 445f931ca1364bb193759d9c6e4ebac9 P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:03.060940 8007 log.cc:1196] T 445f931ca1364bb193759d9c6e4ebac9 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/445f931ca1364bb193759d9c6e4ebac9
I20240502 21:27:03.060969 7716 log.cc:1196] T 445f931ca1364bb193759d9c6e4ebac9 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/445f931ca1364bb193759d9c6e4ebac9
I20240502 21:27:03.061383 7716 ts_tablet_manager.cc:1909] T 445f931ca1364bb193759d9c6e4ebac9 P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:03.061390 8007 ts_tablet_manager.cc:1909] T 445f931ca1364bb193759d9c6e4ebac9 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:27:03.062407 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet 445f931ca1364bb193759d9c6e4ebac9 (table rebalance_test_table_16 [id=78a4b23f64324443a6125591bda63185]) successfully deleted
I20240502 21:27:03.062594 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet 445f931ca1364bb193759d9c6e4ebac9 (table rebalance_test_table_16 [id=78a4b23f64324443a6125591bda63185]) successfully deleted
I20240502 21:27:03.137696 4300 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:03.143678 7214 raft_consensus.cc:1261] T 4c5712eac5c549c3bad16072ec699edc P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:03.143733 7349 raft_consensus.cc:1261] T 4c5712eac5c549c3bad16072ec699edc P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:03.144316 8071 consensus_queue.cc:1035] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:03.144604 7832 consensus_queue.cc:1035] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:03.149325 4294 catalog_manager.cc:5576] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719) does not have the latest schema for tablet 4c5712eac5c549c3bad16072ec699edc (table rebalancer_dynamic_table [id=031b7edd28b641bcae41e11bcb6fcb9a]). Expected version 1 got 0
I20240502 21:27:03.149981 8031 tablet.cc:1722] T 4c5712eac5c549c3bad16072ec699edc P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:03.150558 8031 tablet.cc:1620] T 4c5712eac5c549c3bad16072ec699edc P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:03.151916 4300 catalog_manager.cc:5576] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193) does not have the latest schema for tablet 4c5712eac5c549c3bad16072ec699edc (table rebalancer_dynamic_table [id=031b7edd28b641bcae41e11bcb6fcb9a]). Expected version 1 got 0
I20240502 21:27:03.153220 8078 tablet.cc:1722] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:03.153827 8078 tablet.cc:1620] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:03.155308 8032 tablet.cc:1722] T 4c5712eac5c549c3bad16072ec699edc P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:03.155822 8032 tablet.cc:1620] T 4c5712eac5c549c3bad16072ec699edc P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:03.159560 4300 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=031b7edd28b641bcae41e11bcb6fcb9a] alter complete (version 1)
I20240502 21:27:03.199851 8074 raft_consensus.cc:482] T 648263320b474a32bfffa18b1d4a61e3 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:03.200059 8074 raft_consensus.cc:504] T 648263320b474a32bfffa18b1d4a61e3 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:03.201017 8074 leader_election.cc:288] T 648263320b474a32bfffa18b1d4a61e3 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:03.202145 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "648263320b474a32bfffa18b1d4a61e3" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:27:03.204054 7349 raft_consensus.cc:3037] T 648263320b474a32bfffa18b1d4a61e3 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.206718 7947 consensus_queue.cc:1035] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:03.207804 7947 consensus_queue.cc:1035] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:03.208616 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "648263320b474a32bfffa18b1d4a61e3" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:03.209267 7281 leader_election.cc:302] T 648263320b474a32bfffa18b1d4a61e3 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0456c5e55a542128688904e003d403c; no voters: 31a72286397c4edc9a63eca3412be168, 73a1f2ca1f544de9b0e3b37015f85328
I20240502 21:27:03.211956 8074 raft_consensus.cc:2726] T 648263320b474a32bfffa18b1d4a61e3 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:27:03.251458 8010 consensus_queue.cc:1035] T 171a71f8620c44909b78cb7b35cfed5e P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:03.256579 8010 consensus_queue.cc:1035] T 171a71f8620c44909b78cb7b35cfed5e P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:03.266115 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:03.274395 8078 tablet.cc:1722] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:03.274927 8078 tablet.cc:1620] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:03.275048 8031 tablet.cc:1722] T 4c5712eac5c549c3bad16072ec699edc P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:03.275563 8031 tablet.cc:1620] T 4c5712eac5c549c3bad16072ec699edc P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:03.275774 8032 tablet.cc:1722] T 4c5712eac5c549c3bad16072ec699edc P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:03.276296 8032 tablet.cc:1620] T 4c5712eac5c549c3bad16072ec699edc P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:03.284627 4294 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=031b7edd28b641bcae41e11bcb6fcb9a] alter complete (version 2)
I20240502 21:27:03.331043 7705 consensus_queue.cc:1035] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:03.337229 7534 consensus_queue.cc:1035] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:03.387758 4298 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:03.395628 8078 tablet.cc:1722] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:03.396323 8031 tablet.cc:1722] T 4c5712eac5c549c3bad16072ec699edc P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:03.396728 8032 tablet.cc:1722] T 4c5712eac5c549c3bad16072ec699edc P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:03.404750 4294 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=031b7edd28b641bcae41e11bcb6fcb9a] alter complete (version 3)
I20240502 21:27:03.440639 4294 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:03.449180 7194 tablet_service.cc:1467] Processing CreateTablet for tablet 26a00b5700a54672b7389e87c2a4dfbd (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=d5866bdd67f64fe5bbeebaed4bd1f67e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:03.449294 7328 tablet_service.cc:1467] Processing CreateTablet for tablet 26a00b5700a54672b7389e87c2a4dfbd (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=d5866bdd67f64fe5bbeebaed4bd1f67e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:03.449645 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26a00b5700a54672b7389e87c2a4dfbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.449561 4555 tablet_service.cc:1467] Processing CreateTablet for tablet 26a00b5700a54672b7389e87c2a4dfbd (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=d5866bdd67f64fe5bbeebaed4bd1f67e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:03.449824 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26a00b5700a54672b7389e87c2a4dfbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.450037 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26a00b5700a54672b7389e87c2a4dfbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.453392 7596 tablet_service.cc:1467] Processing CreateTablet for tablet beaa535c1dbe48b48613582d65e751ae (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=d5866bdd67f64fe5bbeebaed4bd1f67e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:03.453851 7596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beaa535c1dbe48b48613582d65e751ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.457310 7195 tablet_service.cc:1467] Processing CreateTablet for tablet beaa535c1dbe48b48613582d65e751ae (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=d5866bdd67f64fe5bbeebaed4bd1f67e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:03.457898 7195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beaa535c1dbe48b48613582d65e751ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.460014 8090 tablet_bootstrap.cc:492] T 26a00b5700a54672b7389e87c2a4dfbd P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:03.462383 8092 tablet_bootstrap.cc:492] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:03.462711 8091 tablet_bootstrap.cc:492] T 26a00b5700a54672b7389e87c2a4dfbd P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:03.463761 8090 tablet_bootstrap.cc:654] T 26a00b5700a54672b7389e87c2a4dfbd P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.464126 8092 tablet_bootstrap.cc:654] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.464293 8091 tablet_bootstrap.cc:654] T 26a00b5700a54672b7389e87c2a4dfbd P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.466734 8092 tablet_bootstrap.cc:492] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:03.466871 8092 ts_tablet_manager.cc:1367] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:27:03.467747 8092 raft_consensus.cc:348] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:03.468042 8092 raft_consensus.cc:374] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.468150 8092 raft_consensus.cc:729] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.468494 8092 consensus_queue.cc:260] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:03.469223 8092 ts_tablet_manager.cc:1398] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:03.470741 8091 tablet_bootstrap.cc:492] T 26a00b5700a54672b7389e87c2a4dfbd P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:03.470948 8091 ts_tablet_manager.cc:1367] T 26a00b5700a54672b7389e87c2a4dfbd P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20240502 21:27:03.471906 8091 raft_consensus.cc:348] T 26a00b5700a54672b7389e87c2a4dfbd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:03.472077 8091 raft_consensus.cc:374] T 26a00b5700a54672b7389e87c2a4dfbd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.472203 8091 raft_consensus.cc:729] T 26a00b5700a54672b7389e87c2a4dfbd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.472182 4421 tablet_service.cc:1467] Processing CreateTablet for tablet beaa535c1dbe48b48613582d65e751ae (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=d5866bdd67f64fe5bbeebaed4bd1f67e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:03.472481 8091 consensus_queue.cc:260] T 26a00b5700a54672b7389e87c2a4dfbd P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:03.472791 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beaa535c1dbe48b48613582d65e751ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.473796 8091 ts_tablet_manager.cc:1398] T 26a00b5700a54672b7389e87c2a4dfbd P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:03.476877 7328 tablet_service.cc:1467] Processing CreateTablet for tablet 9a36933eaf04449faad747e5f44652b9 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=d5866bdd67f64fe5bbeebaed4bd1f67e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:03.477339 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a36933eaf04449faad747e5f44652b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.478438 7595 tablet_service.cc:1467] Processing CreateTablet for tablet 9a36933eaf04449faad747e5f44652b9 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=d5866bdd67f64fe5bbeebaed4bd1f67e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:03.478936 7595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a36933eaf04449faad747e5f44652b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.484652 8092 tablet_bootstrap.cc:492] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:03.485754 7464 tablet_service.cc:1467] Processing CreateTablet for tablet 9a36933eaf04449faad747e5f44652b9 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=d5866bdd67f64fe5bbeebaed4bd1f67e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:03.486243 7464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a36933eaf04449faad747e5f44652b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.486763 8092 tablet_bootstrap.cc:654] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.490063 8090 tablet_bootstrap.cc:492] T 26a00b5700a54672b7389e87c2a4dfbd P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:03.490284 8090 ts_tablet_manager.cc:1367] T 26a00b5700a54672b7389e87c2a4dfbd P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.009s
I20240502 21:27:03.491333 8090 raft_consensus.cc:348] T 26a00b5700a54672b7389e87c2a4dfbd P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:03.491624 8090 raft_consensus.cc:374] T 26a00b5700a54672b7389e87c2a4dfbd P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.491740 8090 raft_consensus.cc:729] T 26a00b5700a54672b7389e87c2a4dfbd P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.492081 8090 consensus_queue.cc:260] T 26a00b5700a54672b7389e87c2a4dfbd P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:03.493093 8090 ts_tablet_manager.cc:1398] T 26a00b5700a54672b7389e87c2a4dfbd P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:03.493434 8090 tablet_bootstrap.cc:492] T beaa535c1dbe48b48613582d65e751ae P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:03.495730 8090 tablet_bootstrap.cc:654] T beaa535c1dbe48b48613582d65e751ae P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.497578 8092 tablet_bootstrap.cc:492] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:03.497817 8092 ts_tablet_manager.cc:1367] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20240502 21:27:03.499042 8092 raft_consensus.cc:348] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:03.499351 8092 raft_consensus.cc:374] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.499491 8092 raft_consensus.cc:729] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.499837 8092 consensus_queue.cc:260] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:03.500687 8092 ts_tablet_manager.cc:1398] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:03.503562 8090 tablet_bootstrap.cc:492] T beaa535c1dbe48b48613582d65e751ae P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:03.503872 8090 ts_tablet_manager.cc:1367] T beaa535c1dbe48b48613582d65e751ae P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20240502 21:27:03.504762 8090 raft_consensus.cc:348] T beaa535c1dbe48b48613582d65e751ae P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:03.505082 8090 raft_consensus.cc:374] T beaa535c1dbe48b48613582d65e751ae P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.505230 8090 raft_consensus.cc:729] T beaa535c1dbe48b48613582d65e751ae P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.505604 8090 consensus_queue.cc:260] T beaa535c1dbe48b48613582d65e751ae P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:03.507148 8090 ts_tablet_manager.cc:1398] T beaa535c1dbe48b48613582d65e751ae P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:03.508293 4294 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:03.508548 4294 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:03.508777 8099 tablet_bootstrap.cc:492] T beaa535c1dbe48b48613582d65e751ae P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:03.513407 8096 tablet_bootstrap.cc:492] T beaa535c1dbe48b48613582d65e751ae P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:03.515218 8096 tablet_bootstrap.cc:654] T beaa535c1dbe48b48613582d65e751ae P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.516127 8100 tablet_bootstrap.cc:492] T 9a36933eaf04449faad747e5f44652b9 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:03.516127 4294 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 4c5712eac5c549c3bad16072ec699edc
I20240502 21:27:03.517194 7328 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c5712eac5c549c3bad16072ec699edc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:03 UTC) from {username='slave'} at 127.0.0.1:47078
I20240502 21:27:03.517968 7195 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c5712eac5c549c3bad16072ec699edc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:03 UTC) from {username='slave'} at 127.0.0.1:44652
I20240502 21:27:03.519647 7464 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c5712eac5c549c3bad16072ec699edc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:03 UTC) from {username='slave'} at 127.0.0.1:37182
I20240502 21:27:03.521071 8099 tablet_bootstrap.cc:654] T beaa535c1dbe48b48613582d65e751ae P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.521986 8100 tablet_bootstrap.cc:654] T 9a36933eaf04449faad747e5f44652b9 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.531721 8104 tablet_replica.cc:330] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:27:03.532894 8104 raft_consensus.cc:2229] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:03.533032 8096 tablet_bootstrap.cc:492] T beaa535c1dbe48b48613582d65e751ae P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:03.534162 8096 ts_tablet_manager.cc:1367] T beaa535c1dbe48b48613582d65e751ae P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.005s
I20240502 21:27:03.534730 8103 tablet_replica.cc:330] T 4c5712eac5c549c3bad16072ec699edc P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:27:03.535604 8104 raft_consensus.cc:2258] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:03.536644 8096 raft_consensus.cc:348] T beaa535c1dbe48b48613582d65e751ae P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:03.537017 8096 raft_consensus.cc:374] T beaa535c1dbe48b48613582d65e751ae P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.537153 8104 ts_tablet_manager.cc:1875] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:03.537142 8096 raft_consensus.cc:729] T beaa535c1dbe48b48613582d65e751ae P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.537521 8096 consensus_queue.cc:260] T beaa535c1dbe48b48613582d65e751ae P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:03.538216 8102 tablet_replica.cc:330] T 4c5712eac5c549c3bad16072ec699edc P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:27:03.538390 8102 raft_consensus.cc:2229] T 4c5712eac5c549c3bad16072ec699edc P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:03.540655 8103 raft_consensus.cc:2229] T 4c5712eac5c549c3bad16072ec699edc P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:03.540676 8096 ts_tablet_manager.cc:1398] T beaa535c1dbe48b48613582d65e751ae P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 21:27:03.540949 8103 raft_consensus.cc:2258] T 4c5712eac5c549c3bad16072ec699edc P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:03.541008 8096 tablet_bootstrap.cc:492] T 9a36933eaf04449faad747e5f44652b9 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:03.541566 8102 raft_consensus.cc:2258] T 4c5712eac5c549c3bad16072ec699edc P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:03.543299 8096 tablet_bootstrap.cc:654] T 9a36933eaf04449faad747e5f44652b9 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.543596 8099 tablet_bootstrap.cc:492] T beaa535c1dbe48b48613582d65e751ae P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:03.543864 8099 ts_tablet_manager.cc:1367] T beaa535c1dbe48b48613582d65e751ae P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.000s
I20240502 21:27:03.544195 8102 ts_tablet_manager.cc:1875] T 4c5712eac5c549c3bad16072ec699edc P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:03.544807 8099 raft_consensus.cc:348] T beaa535c1dbe48b48613582d65e751ae P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:03.545037 8099 raft_consensus.cc:374] T beaa535c1dbe48b48613582d65e751ae P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.545157 8099 raft_consensus.cc:729] T beaa535c1dbe48b48613582d65e751ae P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.545403 8099 consensus_queue.cc:260] T beaa535c1dbe48b48613582d65e751ae P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:03.546147 4294 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240502 21:27:03.546360 4294 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240502 21:27:03.547303 8100 tablet_bootstrap.cc:492] T 9a36933eaf04449faad747e5f44652b9 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:03.547545 8100 ts_tablet_manager.cc:1367] T 9a36933eaf04449faad747e5f44652b9 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.000s
I20240502 21:27:03.549540 8100 raft_consensus.cc:348] T 9a36933eaf04449faad747e5f44652b9 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:03.549829 8100 raft_consensus.cc:374] T 9a36933eaf04449faad747e5f44652b9 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.549978 8100 raft_consensus.cc:729] T 9a36933eaf04449faad747e5f44652b9 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.550267 8100 consensus_queue.cc:260] T 9a36933eaf04449faad747e5f44652b9 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:03.550712 4294 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet b16b4fe400714cb99c0b646ca0566329
I20240502 21:27:03.550752 8104 ts_tablet_manager.cc:1888] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:03.550889 8104 log.cc:1196] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/4c5712eac5c549c3bad16072ec699edc
I20240502 21:27:03.551414 8104 ts_tablet_manager.cc:1909] T 4c5712eac5c549c3bad16072ec699edc P 31a72286397c4edc9a63eca3412be168: Deleting consensus metadata
I20240502 21:27:03.551442 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet b16b4fe400714cb99c0b646ca0566329 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:03 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:03.551892 7716 tablet_replica.cc:330] T b16b4fe400714cb99c0b646ca0566329 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:03.551875 8100 ts_tablet_manager.cc:1398] T 9a36933eaf04449faad747e5f44652b9 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 21:27:03.551873 4818 tablet_service.cc:1514] Processing DeleteTablet for tablet b16b4fe400714cb99c0b646ca0566329 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:03 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:27:03.552098 7716 raft_consensus.cc:2229] T b16b4fe400714cb99c0b646ca0566329 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:03.552157 8007 tablet_replica.cc:330] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:27:03.552387 8007 raft_consensus.cc:2229] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:03.552735 8103 ts_tablet_manager.cc:1875] T 4c5712eac5c549c3bad16072ec699edc P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:03.552745 8007 raft_consensus.cc:2258] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:03.553519 8007 ts_tablet_manager.cc:1875] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:03.553795 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet 4c5712eac5c549c3bad16072ec699edc (table rebalancer_dynamic_table_new_name [id=031b7edd28b641bcae41e11bcb6fcb9a]) successfully deleted
I20240502 21:27:03.555018 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet b16b4fe400714cb99c0b646ca0566329 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:03 UTC) from {username='slave'} at 127.0.0.1:60018
I20240502 21:27:03.555608 8107 tablet_replica.cc:330] T b16b4fe400714cb99c0b646ca0566329 P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:27:03.555809 8107 raft_consensus.cc:2229] T b16b4fe400714cb99c0b646ca0566329 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:03.556845 4294 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 3131b965571144fe9480d59f7607d697
I20240502 21:27:03.556063 8107 raft_consensus.cc:2258] T b16b4fe400714cb99c0b646ca0566329 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:03.557611 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet 3131b965571144fe9480d59f7607d697 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:03 UTC) from {username='slave'} at 127.0.0.1:44764
I20240502 21:27:03.558208 8107 ts_tablet_manager.cc:1875] T b16b4fe400714cb99c0b646ca0566329 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:03.558347 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet 3131b965571144fe9480d59f7607d697 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:03 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:03.558578 4294 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet d43765656b5e415886b053e7866dc3b8
I20240502 21:27:03.558667 4818 tablet_service.cc:1514] Processing DeleteTablet for tablet 3131b965571144fe9480d59f7607d697 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:03 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:27:03.559329 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet d43765656b5e415886b053e7866dc3b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:03 UTC) from {username='slave'} at 127.0.0.1:60018
I20240502 21:27:03.561087 4818 tablet_service.cc:1514] Processing DeleteTablet for tablet d43765656b5e415886b053e7866dc3b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:03 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:27:03.561468 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet d43765656b5e415886b053e7866dc3b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:03 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:03.562320 8007 ts_tablet_manager.cc:1888] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:03.562495 8007 log.cc:1196] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/b16b4fe400714cb99c0b646ca0566329
I20240502 21:27:03.562903 8007 ts_tablet_manager.cc:1909] T b16b4fe400714cb99c0b646ca0566329 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:27:03.563830 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet b16b4fe400714cb99c0b646ca0566329 (table rebalance_test_table_18 [id=41418e80a1734c7e84d8140b5c8fb2c3]) successfully deleted
I20240502 21:27:03.564738 7716 raft_consensus.cc:2258] T b16b4fe400714cb99c0b646ca0566329 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:03.566287 8099 ts_tablet_manager.cc:1398] T beaa535c1dbe48b48613582d65e751ae P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240502 21:27:03.566749 8007 tablet_replica.cc:330] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:27:03.566963 8107 ts_tablet_manager.cc:1888] T b16b4fe400714cb99c0b646ca0566329 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:03.566989 8007 raft_consensus.cc:2229] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:03.566552 7716 ts_tablet_manager.cc:1875] T b16b4fe400714cb99c0b646ca0566329 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:03.567165 8107 log.cc:1196] T b16b4fe400714cb99c0b646ca0566329 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/b16b4fe400714cb99c0b646ca0566329
I20240502 21:27:03.567694 8107 ts_tablet_manager.cc:1909] T b16b4fe400714cb99c0b646ca0566329 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting consensus metadata
I20240502 21:27:03.568931 8107 tablet_replica.cc:330] T d43765656b5e415886b053e7866dc3b8 P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:27:03.568992 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet b16b4fe400714cb99c0b646ca0566329 (table rebalance_test_table_18 [id=41418e80a1734c7e84d8140b5c8fb2c3]) successfully deleted
I20240502 21:27:03.569221 8107 raft_consensus.cc:2229] T d43765656b5e415886b053e7866dc3b8 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:03.569451 8107 raft_consensus.cc:2258] T d43765656b5e415886b053e7866dc3b8 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:03.570789 8107 ts_tablet_manager.cc:1875] T d43765656b5e415886b053e7866dc3b8 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:03.577287 7716 ts_tablet_manager.cc:1888] T b16b4fe400714cb99c0b646ca0566329 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:03.577474 7716 log.cc:1196] T b16b4fe400714cb99c0b646ca0566329 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/b16b4fe400714cb99c0b646ca0566329
I20240502 21:27:03.577952 7716 ts_tablet_manager.cc:1909] T b16b4fe400714cb99c0b646ca0566329 P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:03.578783 8107 ts_tablet_manager.cc:1888] T d43765656b5e415886b053e7866dc3b8 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:03.579022 8107 log.cc:1196] T d43765656b5e415886b053e7866dc3b8 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/d43765656b5e415886b053e7866dc3b8
I20240502 21:27:03.578969 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet b16b4fe400714cb99c0b646ca0566329 (table rebalance_test_table_18 [id=41418e80a1734c7e84d8140b5c8fb2c3]) successfully deleted
I20240502 21:27:03.579545 8107 ts_tablet_manager.cc:1909] T d43765656b5e415886b053e7866dc3b8 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting consensus metadata
I20240502 21:27:03.580060 8007 raft_consensus.cc:2258] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:03.580814 7716 tablet_replica.cc:330] T 3131b965571144fe9480d59f7607d697 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:03.580868 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet d43765656b5e415886b053e7866dc3b8 (table rebalance_test_table_18 [id=41418e80a1734c7e84d8140b5c8fb2c3]) successfully deleted
I20240502 21:27:03.581084 7716 raft_consensus.cc:2229] T 3131b965571144fe9480d59f7607d697 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:03.581337 7716 raft_consensus.cc:2258] T 3131b965571144fe9480d59f7607d697 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:03.582037 7716 ts_tablet_manager.cc:1875] T 3131b965571144fe9480d59f7607d697 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:03.583736 8007 ts_tablet_manager.cc:1875] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:03.584128 8074 raft_consensus.cc:482] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:03.584280 8074 raft_consensus.cc:504] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:03.585002 8074 leader_election.cc:288] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:27:03.585676 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a00b5700a54672b7389e87c2a4dfbd" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:03.585835 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a00b5700a54672b7389e87c2a4dfbd" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:27:03.585965 7214 raft_consensus.cc:2454] T 26a00b5700a54672b7389e87c2a4dfbd P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 0.
I20240502 21:27:03.586032 4574 raft_consensus.cc:2454] T 26a00b5700a54672b7389e87c2a4dfbd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 0.
I20240502 21:27:03.586232 8103 ts_tablet_manager.cc:1888] T 4c5712eac5c549c3bad16072ec699edc P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:03.586359 8103 log.cc:1196] T 4c5712eac5c549c3bad16072ec699edc P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/4c5712eac5c549c3bad16072ec699edc
I20240502 21:27:03.586488 7283 leader_election.cc:302] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:03.587255 8074 raft_consensus.cc:2781] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:03.587383 8074 raft_consensus.cc:482] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:03.587481 8074 raft_consensus.cc:3037] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.587687 8109 tablet_replica.cc:330] T 3131b965571144fe9480d59f7607d697 P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:27:03.589129 8102 ts_tablet_manager.cc:1888] T 4c5712eac5c549c3bad16072ec699edc P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:03.589255 8102 log.cc:1196] T 4c5712eac5c549c3bad16072ec699edc P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/4c5712eac5c549c3bad16072ec699edc
I20240502 21:27:03.589805 8102 ts_tablet_manager.cc:1909] T 4c5712eac5c549c3bad16072ec699edc P d0456c5e55a542128688904e003d403c: Deleting consensus metadata
W20240502 21:27:03.590210 4378 consensus_peers.cc:485] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d43765656b5e415886b053e7866dc3b8. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:03.590369 8074 raft_consensus.cc:504] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:03.587920 8109 raft_consensus.cc:2229] T 3131b965571144fe9480d59f7607d697 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:03.590838 8109 raft_consensus.cc:2258] T 3131b965571144fe9480d59f7607d697 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:03.591727 8109 ts_tablet_manager.cc:1875] T 3131b965571144fe9480d59f7607d697 P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:03.591729 8007 ts_tablet_manager.cc:1888] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:03.591866 8007 log.cc:1196] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/3131b965571144fe9480d59f7607d697
I20240502 21:27:03.591912 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a00b5700a54672b7389e87c2a4dfbd" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:03.592137 7214 raft_consensus.cc:3037] T 26a00b5700a54672b7389e87c2a4dfbd P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.592329 8007 ts_tablet_manager.cc:1909] T 3131b965571144fe9480d59f7607d697 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:27:03.593335 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet 3131b965571144fe9480d59f7607d697 (table rebalance_test_table_18 [id=41418e80a1734c7e84d8140b5c8fb2c3]) successfully deleted
I20240502 21:27:03.594775 7214 raft_consensus.cc:2454] T 26a00b5700a54672b7389e87c2a4dfbd P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
I20240502 21:27:03.595554 7283 leader_election.cc:302] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:03.595929 7953 raft_consensus.cc:2781] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:03.596745 8103 ts_tablet_manager.cc:1909] T 4c5712eac5c549c3bad16072ec699edc P 39981a8dfe94462b9a73ba489af4c714: Deleting consensus metadata
I20240502 21:27:03.597174 8096 tablet_bootstrap.cc:492] T 9a36933eaf04449faad747e5f44652b9 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:03.597136 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet 4c5712eac5c549c3bad16072ec699edc (table rebalancer_dynamic_table_new_name [id=031b7edd28b641bcae41e11bcb6fcb9a]) successfully deleted
I20240502 21:27:03.597366 8096 ts_tablet_manager.cc:1367] T 9a36933eaf04449faad747e5f44652b9 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.005s
I20240502 21:27:03.598242 8096 raft_consensus.cc:348] T 9a36933eaf04449faad747e5f44652b9 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:03.598474 8096 raft_consensus.cc:374] T 9a36933eaf04449faad747e5f44652b9 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.598587 8096 raft_consensus.cc:729] T 9a36933eaf04449faad747e5f44652b9 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.598807 7716 ts_tablet_manager.cc:1888] T 3131b965571144fe9480d59f7607d697 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:03.598964 7716 log.cc:1196] T 3131b965571144fe9480d59f7607d697 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/3131b965571144fe9480d59f7607d697
I20240502 21:27:03.598851 8096 consensus_queue.cc:260] T 9a36933eaf04449faad747e5f44652b9 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:03.599298 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet 4c5712eac5c549c3bad16072ec699edc (table rebalancer_dynamic_table_new_name [id=031b7edd28b641bcae41e11bcb6fcb9a]) successfully deleted
I20240502 21:27:03.599432 7716 ts_tablet_manager.cc:1909] T 3131b965571144fe9480d59f7607d697 P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:03.599612 7953 raft_consensus.cc:686] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Becoming Leader. State: Replica: d0456c5e55a542128688904e003d403c, State: Running, Role: LEADER
I20240502 21:27:03.599918 7953 consensus_queue.cc:237] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:03.600162 8096 ts_tablet_manager.cc:1398] T 9a36933eaf04449faad747e5f44652b9 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240502 21:27:03.600870 7716 tablet_replica.cc:330] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:03.601102 7716 raft_consensus.cc:2229] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:03.601233 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet 3131b965571144fe9480d59f7607d697 (table rebalance_test_table_18 [id=41418e80a1734c7e84d8140b5c8fb2c3]) successfully deleted
I20240502 21:27:03.601320 8109 ts_tablet_manager.cc:1888] T 3131b965571144fe9480d59f7607d697 P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:03.601442 8109 log.cc:1196] T 3131b965571144fe9480d59f7607d697 P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/3131b965571144fe9480d59f7607d697
I20240502 21:27:03.601480 7716 raft_consensus.cc:2258] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:03.601989 8109 ts_tablet_manager.cc:1909] T 3131b965571144fe9480d59f7607d697 P 398c3c8f3195492c900439ad5e1931bb: Deleting consensus metadata
I20240502 21:27:03.602219 7716 ts_tablet_manager.cc:1875] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:03.602916 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet 3131b965571144fe9480d59f7607d697 (table rebalance_test_table_18 [id=41418e80a1734c7e84d8140b5c8fb2c3]) successfully deleted
I20240502 21:27:03.603240 8007 tablet_replica.cc:330] T d43765656b5e415886b053e7866dc3b8 P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:27:03.603468 8007 raft_consensus.cc:2229] T d43765656b5e415886b053e7866dc3b8 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:03.603708 8007 raft_consensus.cc:2258] T d43765656b5e415886b053e7866dc3b8 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:03.604470 8007 ts_tablet_manager.cc:1875] T d43765656b5e415886b053e7866dc3b8 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:03.607503 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a00b5700a54672b7389e87c2a4dfbd" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:27:03.607756 4574 raft_consensus.cc:3037] T 26a00b5700a54672b7389e87c2a4dfbd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.608013 4292 catalog_manager.cc:5484] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c reported cstate change: term changed from 0 to 1, leader changed from <none> to d0456c5e55a542128688904e003d403c (127.29.82.70). New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:03.609161 7716 ts_tablet_manager.cc:1888] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:03.609302 7716 log.cc:1196] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/d43765656b5e415886b053e7866dc3b8
I20240502 21:27:03.609771 7716 ts_tablet_manager.cc:1909] T d43765656b5e415886b053e7866dc3b8 P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:03.610615 4574 raft_consensus.cc:2454] T 26a00b5700a54672b7389e87c2a4dfbd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
I20240502 21:27:03.611160 8007 ts_tablet_manager.cc:1888] T d43765656b5e415886b053e7866dc3b8 P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:27:03.611248 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet d43765656b5e415886b053e7866dc3b8 (table rebalance_test_table_18 [id=41418e80a1734c7e84d8140b5c8fb2c3]) successfully deleted
I20240502 21:27:03.611318 8007 log.cc:1196] T d43765656b5e415886b053e7866dc3b8 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/d43765656b5e415886b053e7866dc3b8
I20240502 21:27:03.611797 8007 ts_tablet_manager.cc:1909] T d43765656b5e415886b053e7866dc3b8 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:27:03.611968 8074 leader_election.cc:288] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Requested vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:27:03.612771 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet d43765656b5e415886b053e7866dc3b8 (table rebalance_test_table_18 [id=41418e80a1734c7e84d8140b5c8fb2c3]) successfully deleted
I20240502 21:27:03.618613 4292 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:03.625319 7596 tablet_service.cc:1467] Processing CreateTablet for tablet 7ca0a9da86174e388f7e428fc3ab2705 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=23a138e169b341daacc3908cd46a1eb7]), partition=
I20240502 21:27:03.625720 7596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ca0a9da86174e388f7e428fc3ab2705. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.625730 7195 tablet_service.cc:1467] Processing CreateTablet for tablet 7ca0a9da86174e388f7e428fc3ab2705 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=23a138e169b341daacc3908cd46a1eb7]), partition=
I20240502 21:27:03.626085 7195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ca0a9da86174e388f7e428fc3ab2705. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.628808 7328 tablet_service.cc:1467] Processing CreateTablet for tablet 7ca0a9da86174e388f7e428fc3ab2705 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=23a138e169b341daacc3908cd46a1eb7]), partition=
I20240502 21:27:03.629182 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ca0a9da86174e388f7e428fc3ab2705. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.630702 8090 tablet_bootstrap.cc:492] T 7ca0a9da86174e388f7e428fc3ab2705 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:03.631531 8096 tablet_bootstrap.cc:492] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:03.632030 8090 tablet_bootstrap.cc:654] T 7ca0a9da86174e388f7e428fc3ab2705 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.633117 8096 tablet_bootstrap.cc:654] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.635886 8090 tablet_bootstrap.cc:492] T 7ca0a9da86174e388f7e428fc3ab2705 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:03.636025 8090 ts_tablet_manager.cc:1367] T 7ca0a9da86174e388f7e428fc3ab2705 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240502 21:27:03.636415 8092 tablet_bootstrap.cc:492] T 7ca0a9da86174e388f7e428fc3ab2705 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:03.636440 8096 tablet_bootstrap.cc:492] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:03.636580 8096 ts_tablet_manager.cc:1367] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20240502 21:27:03.636818 8090 raft_consensus.cc:348] T 7ca0a9da86174e388f7e428fc3ab2705 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:03.637014 8090 raft_consensus.cc:374] T 7ca0a9da86174e388f7e428fc3ab2705 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.637104 8090 raft_consensus.cc:729] T 7ca0a9da86174e388f7e428fc3ab2705 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.637357 8096 raft_consensus.cc:348] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:03.637336 8090 consensus_queue.cc:260] T 7ca0a9da86174e388f7e428fc3ab2705 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:03.637540 8096 raft_consensus.cc:374] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.637631 8096 raft_consensus.cc:729] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.637749 8092 tablet_bootstrap.cc:654] T 7ca0a9da86174e388f7e428fc3ab2705 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.637933 8090 ts_tablet_manager.cc:1398] T 7ca0a9da86174e388f7e428fc3ab2705 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:03.637878 8096 consensus_queue.cc:260] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:03.639227 8096 ts_tablet_manager.cc:1398] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 21:27:03.640551 8092 tablet_bootstrap.cc:492] T 7ca0a9da86174e388f7e428fc3ab2705 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:03.640698 8092 ts_tablet_manager.cc:1367] T 7ca0a9da86174e388f7e428fc3ab2705 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:27:03.641286 8092 raft_consensus.cc:348] T 7ca0a9da86174e388f7e428fc3ab2705 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:03.641479 8092 raft_consensus.cc:374] T 7ca0a9da86174e388f7e428fc3ab2705 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.641587 8092 raft_consensus.cc:729] T 7ca0a9da86174e388f7e428fc3ab2705 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.642269 8092 consensus_queue.cc:260] T 7ca0a9da86174e388f7e428fc3ab2705 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:03.642997 8092 ts_tablet_manager.cc:1398] T 7ca0a9da86174e388f7e428fc3ab2705 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:03.699360 7942 raft_consensus.cc:482] T beaa535c1dbe48b48613582d65e751ae P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:03.699599 7942 raft_consensus.cc:504] T beaa535c1dbe48b48613582d65e751ae P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:03.700445 7942 leader_election.cc:288] T beaa535c1dbe48b48613582d65e751ae P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:03.701157 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beaa535c1dbe48b48613582d65e751ae" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:03.701488 7214 raft_consensus.cc:2454] T beaa535c1dbe48b48613582d65e751ae P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:03.702451 7551 leader_election.cc:302] T beaa535c1dbe48b48613582d65e751ae P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:03.702772 7942 raft_consensus.cc:2781] T beaa535c1dbe48b48613582d65e751ae P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:03.702884 7942 raft_consensus.cc:482] T beaa535c1dbe48b48613582d65e751ae P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:03.702976 7942 raft_consensus.cc:3037] T beaa535c1dbe48b48613582d65e751ae P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.705499 7942 raft_consensus.cc:504] T beaa535c1dbe48b48613582d65e751ae P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:03.706178 7942 leader_election.cc:288] T beaa535c1dbe48b48613582d65e751ae P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:03.706719 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beaa535c1dbe48b48613582d65e751ae" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:03.706940 7214 raft_consensus.cc:3037] T beaa535c1dbe48b48613582d65e751ae P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.709290 7214 raft_consensus.cc:2454] T beaa535c1dbe48b48613582d65e751ae P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:03.709231 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beaa535c1dbe48b48613582d65e751ae" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:27:03.709347 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beaa535c1dbe48b48613582d65e751ae" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:27:03.709522 4443 raft_consensus.cc:2454] T beaa535c1dbe48b48613582d65e751ae P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:03.709785 7551 leader_election.cc:302] T beaa535c1dbe48b48613582d65e751ae P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:03.710085 7942 raft_consensus.cc:2781] T beaa535c1dbe48b48613582d65e751ae P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:03.710510 7942 raft_consensus.cc:686] T beaa535c1dbe48b48613582d65e751ae P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Becoming Leader. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Running, Role: LEADER
I20240502 21:27:03.710875 7942 consensus_queue.cc:237] T beaa535c1dbe48b48613582d65e751ae P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:03.713958 4292 catalog_manager.cc:5484] T beaa535c1dbe48b48613582d65e751ae P 0ddd486109f5434cace13d60d633ac04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ddd486109f5434cace13d60d633ac04 (127.29.82.72). New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:03.844691 7942 raft_consensus.cc:482] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:03.844906 7942 raft_consensus.cc:504] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:03.845621 7942 leader_election.cc:288] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:03.846093 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ca0a9da86174e388f7e428fc3ab2705" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:03.846119 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ca0a9da86174e388f7e428fc3ab2705" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:03.846344 7350 raft_consensus.cc:2454] T 7ca0a9da86174e388f7e428fc3ab2705 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:03.846415 7214 raft_consensus.cc:2454] T 7ca0a9da86174e388f7e428fc3ab2705 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:03.846928 7551 leader_election.cc:302] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:03.847250 7942 raft_consensus.cc:2781] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:03.847352 7942 raft_consensus.cc:482] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:03.847447 7942 raft_consensus.cc:3037] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.849615 7942 raft_consensus.cc:504] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:03.850175 7942 leader_election.cc:288] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:03.850509 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ca0a9da86174e388f7e428fc3ab2705" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:03.850665 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ca0a9da86174e388f7e428fc3ab2705" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:03.850741 7350 raft_consensus.cc:3037] T 7ca0a9da86174e388f7e428fc3ab2705 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.850880 7214 raft_consensus.cc:3037] T 7ca0a9da86174e388f7e428fc3ab2705 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.853014 7214 raft_consensus.cc:2454] T 7ca0a9da86174e388f7e428fc3ab2705 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:03.853451 7350 raft_consensus.cc:2454] T 7ca0a9da86174e388f7e428fc3ab2705 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:03.853446 7551 leader_election.cc:302] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:03.853942 7942 raft_consensus.cc:2781] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:03.854116 7942 raft_consensus.cc:686] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Becoming Leader. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Running, Role: LEADER
I20240502 21:27:03.854434 7942 consensus_queue.cc:237] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:03.857630 4292 catalog_manager.cc:5484] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ddd486109f5434cace13d60d633ac04 (127.29.82.72). New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:03.872268 8074 raft_consensus.cc:482] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:03.872418 8074 raft_consensus.cc:504] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:03.873080 8074 leader_election.cc:288] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:03.873539 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a36933eaf04449faad747e5f44652b9" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:03.873582 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a36933eaf04449faad747e5f44652b9" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:03.873777 7484 raft_consensus.cc:2454] T 9a36933eaf04449faad747e5f44652b9 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 0.
I20240502 21:27:03.873808 7618 raft_consensus.cc:2454] T 9a36933eaf04449faad747e5f44652b9 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 0.
I20240502 21:27:03.874217 7281 leader_election.cc:302] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:03.874565 8074 raft_consensus.cc:2781] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:03.874687 8074 raft_consensus.cc:482] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:03.874764 8074 raft_consensus.cc:3037] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.876680 8074 raft_consensus.cc:504] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:03.877210 8074 leader_election.cc:288] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Requested vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:03.877615 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a36933eaf04449faad747e5f44652b9" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:03.877679 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a36933eaf04449faad747e5f44652b9" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:03.877804 7618 raft_consensus.cc:3037] T 9a36933eaf04449faad747e5f44652b9 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.877893 7484 raft_consensus.cc:3037] T 9a36933eaf04449faad747e5f44652b9 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.879729 7618 raft_consensus.cc:2454] T 9a36933eaf04449faad747e5f44652b9 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
I20240502 21:27:03.879786 7484 raft_consensus.cc:2454] T 9a36933eaf04449faad747e5f44652b9 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
I20240502 21:27:03.880111 7282 leader_election.cc:302] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:03.880396 8074 raft_consensus.cc:2781] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:03.880522 8074 raft_consensus.cc:686] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Becoming Leader. State: Replica: d0456c5e55a542128688904e003d403c, State: Running, Role: LEADER
I20240502 21:27:03.880795 8074 consensus_queue.cc:237] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:03.884011 4292 catalog_manager.cc:5484] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c reported cstate change: term changed from 0 to 1, leader changed from <none> to d0456c5e55a542128688904e003d403c (127.29.82.70). New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:27:03.969798 4292 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:03.976310 7350 raft_consensus.cc:1261] T 7ca0a9da86174e388f7e428fc3ab2705 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:03.976388 7214 raft_consensus.cc:1261] T 7ca0a9da86174e388f7e428fc3ab2705 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:03.976923 8118 consensus_queue.cc:1035] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:03.977216 7942 consensus_queue.cc:1035] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:03.986176 4292 catalog_manager.cc:5576] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193) does not have the latest schema for tablet 7ca0a9da86174e388f7e428fc3ab2705 (table rebalancer_dynamic_table [id=23a138e169b341daacc3908cd46a1eb7]). Expected version 1 got 0
I20240502 21:27:03.989428 8128 tablet.cc:1722] T 7ca0a9da86174e388f7e428fc3ab2705 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:03.990072 4292 catalog_manager.cc:5576] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719) does not have the latest schema for tablet 7ca0a9da86174e388f7e428fc3ab2705 (table rebalancer_dynamic_table [id=23a138e169b341daacc3908cd46a1eb7]). Expected version 1 got 0
I20240502 21:27:03.987385 8127 tablet.cc:1722] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:03.990842 4294 catalog_manager.cc:5576] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193) does not have the latest schema for tablet 7ca0a9da86174e388f7e428fc3ab2705 (table rebalancer_dynamic_table [id=23a138e169b341daacc3908cd46a1eb7]). Expected version 1 got 0
I20240502 21:27:03.991340 8127 tablet.cc:1620] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04: MemRowSet was empty: no flush needed.
I20240502 21:27:03.992374 8128 tablet.cc:1620] T 7ca0a9da86174e388f7e428fc3ab2705 P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:03.996073 8130 tablet.cc:1722] T 7ca0a9da86174e388f7e428fc3ab2705 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:03.997406 8130 tablet.cc:1620] T 7ca0a9da86174e388f7e428fc3ab2705 P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:04.001219 4294 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=23a138e169b341daacc3908cd46a1eb7] alter complete (version 1)
I20240502 21:27:04.009370 4294 catalog_manager.cc:5576] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719) does not have the latest schema for tablet 7ca0a9da86174e388f7e428fc3ab2705 (table rebalancer_dynamic_table [id=23a138e169b341daacc3908cd46a1eb7]). Expected version 1 got 0
I20240502 21:27:04.088891 8074 consensus_queue.cc:1035] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:04.093437 7953 consensus_queue.cc:1035] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:04.106248 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:04.112526 4443 raft_consensus.cc:3037] T beaa535c1dbe48b48613582d65e751ae P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.113952 8127 tablet.cc:1722] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:04.114456 8127 tablet.cc:1620] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04: MemRowSet was empty: no flush needed.
I20240502 21:27:04.115437 8128 tablet.cc:1722] T 7ca0a9da86174e388f7e428fc3ab2705 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:04.115932 8128 tablet.cc:1620] T 7ca0a9da86174e388f7e428fc3ab2705 P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:04.120576 8130 tablet.cc:1722] T 7ca0a9da86174e388f7e428fc3ab2705 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:04.120540 7942 consensus_queue.cc:1035] T beaa535c1dbe48b48613582d65e751ae P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:04.121033 8130 tablet.cc:1620] T 7ca0a9da86174e388f7e428fc3ab2705 P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:04.126485 4292 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=23a138e169b341daacc3908cd46a1eb7] alter complete (version 2)
I20240502 21:27:04.134245 8135 consensus_queue.cc:1035] T beaa535c1dbe48b48613582d65e751ae P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:04.229648 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:04.237885 8127 tablet.cc:1722] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:04.238437 8128 tablet.cc:1722] T 7ca0a9da86174e388f7e428fc3ab2705 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:04.238519 8130 tablet.cc:1722] T 7ca0a9da86174e388f7e428fc3ab2705 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:04.246672 4292 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=23a138e169b341daacc3908cd46a1eb7] alter complete (version 3)
I20240502 21:27:04.351027 4292 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:04.351256 4292 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:04.354415 4292 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 7ca0a9da86174e388f7e428fc3ab2705
I20240502 21:27:04.355224 7596 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ca0a9da86174e388f7e428fc3ab2705 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:04 UTC) from {username='slave'} at 127.0.0.1:44444
I20240502 21:27:04.355494 7328 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ca0a9da86174e388f7e428fc3ab2705 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:04 UTC) from {username='slave'} at 127.0.0.1:47078
I20240502 21:27:04.355772 7195 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ca0a9da86174e388f7e428fc3ab2705 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:04 UTC) from {username='slave'} at 127.0.0.1:44652
I20240502 21:27:04.356184 8139 tablet_replica.cc:330] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04: stopping tablet replica
I20240502 21:27:04.356457 8139 raft_consensus.cc:2229] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:04.357091 8139 raft_consensus.cc:2258] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:04.357396 8140 tablet_replica.cc:330] T 7ca0a9da86174e388f7e428fc3ab2705 P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:27:04.357659 8140 raft_consensus.cc:2229] T 7ca0a9da86174e388f7e428fc3ab2705 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:04.357901 8140 raft_consensus.cc:2258] T 7ca0a9da86174e388f7e428fc3ab2705 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:04.358685 8139 ts_tablet_manager.cc:1875] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:04.358740 8140 ts_tablet_manager.cc:1875] T 7ca0a9da86174e388f7e428fc3ab2705 P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:04.365201 8139 ts_tablet_manager.cc:1888] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:04.365590 8139 log.cc:1196] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/wals/7ca0a9da86174e388f7e428fc3ab2705
I20240502 21:27:04.366192 8139 ts_tablet_manager.cc:1909] T 7ca0a9da86174e388f7e428fc3ab2705 P 0ddd486109f5434cace13d60d633ac04: Deleting consensus metadata
I20240502 21:27:04.366814 8140 ts_tablet_manager.cc:1888] T 7ca0a9da86174e388f7e428fc3ab2705 P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:04.367033 8140 log.cc:1196] T 7ca0a9da86174e388f7e428fc3ab2705 P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/7ca0a9da86174e388f7e428fc3ab2705
I20240502 21:27:04.367568 4285 catalog_manager.cc:4830] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): tablet 7ca0a9da86174e388f7e428fc3ab2705 (table rebalancer_dynamic_table_new_name [id=23a138e169b341daacc3908cd46a1eb7]) successfully deleted
I20240502 21:27:04.367681 8140 ts_tablet_manager.cc:1909] T 7ca0a9da86174e388f7e428fc3ab2705 P d0456c5e55a542128688904e003d403c: Deleting consensus metadata
I20240502 21:27:04.369011 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet 7ca0a9da86174e388f7e428fc3ab2705 (table rebalancer_dynamic_table_new_name [id=23a138e169b341daacc3908cd46a1eb7]) successfully deleted
I20240502 21:27:04.369272 8141 tablet_replica.cc:330] T 7ca0a9da86174e388f7e428fc3ab2705 P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:27:04.369444 8141 raft_consensus.cc:2229] T 7ca0a9da86174e388f7e428fc3ab2705 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:04.369637 8141 raft_consensus.cc:2258] T 7ca0a9da86174e388f7e428fc3ab2705 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:04.370371 8141 ts_tablet_manager.cc:1875] T 7ca0a9da86174e388f7e428fc3ab2705 P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:04.375312 8141 ts_tablet_manager.cc:1888] T 7ca0a9da86174e388f7e428fc3ab2705 P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:04.375450 8141 log.cc:1196] T 7ca0a9da86174e388f7e428fc3ab2705 P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/7ca0a9da86174e388f7e428fc3ab2705
I20240502 21:27:04.375917 8141 ts_tablet_manager.cc:1909] T 7ca0a9da86174e388f7e428fc3ab2705 P 39981a8dfe94462b9a73ba489af4c714: Deleting consensus metadata
I20240502 21:27:04.376791 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet 7ca0a9da86174e388f7e428fc3ab2705 (table rebalancer_dynamic_table_new_name [id=23a138e169b341daacc3908cd46a1eb7]) successfully deleted
I20240502 21:27:04.405612 7953 consensus_queue.cc:1035] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:04.410269 8143 consensus_queue.cc:1035] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:27:04.458374 4292 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:04.465204 4555 tablet_service.cc:1467] Processing CreateTablet for tablet c649fa0da35e4d47ba24f45a227f7e0b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e0941c7363ad44e790631804380d445d]), partition=
I20240502 21:27:04.465588 4421 tablet_service.cc:1467] Processing CreateTablet for tablet c649fa0da35e4d47ba24f45a227f7e0b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e0941c7363ad44e790631804380d445d]), partition=
I20240502 21:27:04.465644 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c649fa0da35e4d47ba24f45a227f7e0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.465673 7195 tablet_service.cc:1467] Processing CreateTablet for tablet c649fa0da35e4d47ba24f45a227f7e0b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e0941c7363ad44e790631804380d445d]), partition=
I20240502 21:27:04.465960 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c649fa0da35e4d47ba24f45a227f7e0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.466059 7195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c649fa0da35e4d47ba24f45a227f7e0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.471551 8146 tablet_bootstrap.cc:492] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:04.474324 4300 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:04.476090 8148 tablet_bootstrap.cc:492] T c649fa0da35e4d47ba24f45a227f7e0b P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:04.473977 8146 tablet_bootstrap.cc:654] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.477778 8148 tablet_bootstrap.cc:654] T c649fa0da35e4d47ba24f45a227f7e0b P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.485497 8147 tablet_bootstrap.cc:492] T c649fa0da35e4d47ba24f45a227f7e0b P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:04.485764 8146 tablet_bootstrap.cc:492] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:04.485935 8146 ts_tablet_manager.cc:1367] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.003s
I20240502 21:27:04.486763 8146 raft_consensus.cc:348] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:04.487030 8146 raft_consensus.cc:374] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.487143 8146 raft_consensus.cc:729] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.487459 8146 consensus_queue.cc:260] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:04.487784 8147 tablet_bootstrap.cc:654] T c649fa0da35e4d47ba24f45a227f7e0b P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.488206 8146 ts_tablet_manager.cc:1398] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:04.488128 7328 tablet_service.cc:1467] Processing CreateTablet for tablet 8f579ec914e540b28a1c7c2e42e498e8 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=e508630eb48f45f88993a4b8e7fd2ea6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:04.488607 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f579ec914e540b28a1c7c2e42e498e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.490509 8148 tablet_bootstrap.cc:492] T c649fa0da35e4d47ba24f45a227f7e0b P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:04.490749 8148 ts_tablet_manager.cc:1367] T c649fa0da35e4d47ba24f45a227f7e0b P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240502 21:27:04.490849 7330 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5fdbfa16c946a38a90be15079bda0c (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=e508630eb48f45f88993a4b8e7fd2ea6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:04.491293 7330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5fdbfa16c946a38a90be15079bda0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.491616 8147 tablet_bootstrap.cc:492] T c649fa0da35e4d47ba24f45a227f7e0b P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:04.491674 8148 raft_consensus.cc:348] T c649fa0da35e4d47ba24f45a227f7e0b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:04.491848 8147 ts_tablet_manager.cc:1367] T c649fa0da35e4d47ba24f45a227f7e0b P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240502 21:27:04.491966 8148 raft_consensus.cc:374] T c649fa0da35e4d47ba24f45a227f7e0b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.492100 8148 raft_consensus.cc:729] T c649fa0da35e4d47ba24f45a227f7e0b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.492401 8148 consensus_queue.cc:260] T c649fa0da35e4d47ba24f45a227f7e0b P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:04.492779 8147 raft_consensus.cc:348] T c649fa0da35e4d47ba24f45a227f7e0b P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:04.492995 8148 ts_tablet_manager.cc:1398] T c649fa0da35e4d47ba24f45a227f7e0b P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:04.493052 8147 raft_consensus.cc:374] T c649fa0da35e4d47ba24f45a227f7e0b P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.494204 8147 raft_consensus.cc:729] T c649fa0da35e4d47ba24f45a227f7e0b P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.494478 8147 consensus_queue.cc:260] T c649fa0da35e4d47ba24f45a227f7e0b P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:04.494859 8152 tablet_bootstrap.cc:492] T 8f579ec914e540b28a1c7c2e42e498e8 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:04.495245 8147 ts_tablet_manager.cc:1398] T c649fa0da35e4d47ba24f45a227f7e0b P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:04.495508 7195 tablet_service.cc:1467] Processing CreateTablet for tablet 8f579ec914e540b28a1c7c2e42e498e8 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=e508630eb48f45f88993a4b8e7fd2ea6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:04.496002 7195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f579ec914e540b28a1c7c2e42e498e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.496811 8152 tablet_bootstrap.cc:654] T 8f579ec914e540b28a1c7c2e42e498e8 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.498849 8152 tablet_bootstrap.cc:492] T 8f579ec914e540b28a1c7c2e42e498e8 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:04.498983 8152 ts_tablet_manager.cc:1367] T 8f579ec914e540b28a1c7c2e42e498e8 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:27:04.499835 8152 raft_consensus.cc:348] T 8f579ec914e540b28a1c7c2e42e498e8 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:04.500067 8152 raft_consensus.cc:374] T 8f579ec914e540b28a1c7c2e42e498e8 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.500169 8152 raft_consensus.cc:729] T 8f579ec914e540b28a1c7c2e42e498e8 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.500401 8152 consensus_queue.cc:260] T 8f579ec914e540b28a1c7c2e42e498e8 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:04.500950 8152 ts_tablet_manager.cc:1398] T 8f579ec914e540b28a1c7c2e42e498e8 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:04.501932 7596 tablet_service.cc:1467] Processing CreateTablet for tablet 8f579ec914e540b28a1c7c2e42e498e8 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=e508630eb48f45f88993a4b8e7fd2ea6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:04.502403 7596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f579ec914e540b28a1c7c2e42e498e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.502478 7464 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5fdbfa16c946a38a90be15079bda0c (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=e508630eb48f45f88993a4b8e7fd2ea6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:04.502676 7194 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5fdbfa16c946a38a90be15079bda0c (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=e508630eb48f45f88993a4b8e7fd2ea6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:04.502941 7464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5fdbfa16c946a38a90be15079bda0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.503091 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5fdbfa16c946a38a90be15079bda0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.510624 8146 tablet_bootstrap.cc:492] T 0a5fdbfa16c946a38a90be15079bda0c P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:04.512441 8146 tablet_bootstrap.cc:654] T 0a5fdbfa16c946a38a90be15079bda0c P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.517199 8154 tablet_bootstrap.cc:492] T 8f579ec914e540b28a1c7c2e42e498e8 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:04.518280 8152 tablet_bootstrap.cc:492] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:04.518394 7330 tablet_service.cc:1467] Processing CreateTablet for tablet 357690713cd44eb3b2a297bfd6bbc17d (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=e508630eb48f45f88993a4b8e7fd2ea6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:04.518880 7330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 357690713cd44eb3b2a297bfd6bbc17d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.524058 8152 tablet_bootstrap.cc:654] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.524117 7595 tablet_service.cc:1467] Processing CreateTablet for tablet 357690713cd44eb3b2a297bfd6bbc17d (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=e508630eb48f45f88993a4b8e7fd2ea6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:04.524578 7595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 357690713cd44eb3b2a297bfd6bbc17d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.526582 8146 tablet_bootstrap.cc:492] T 0a5fdbfa16c946a38a90be15079bda0c P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:04.526751 8146 ts_tablet_manager.cc:1367] T 0a5fdbfa16c946a38a90be15079bda0c P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.005s
I20240502 21:27:04.527076 8155 tablet_bootstrap.cc:492] T 0a5fdbfa16c946a38a90be15079bda0c P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:04.527632 8146 raft_consensus.cc:348] T 0a5fdbfa16c946a38a90be15079bda0c P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:04.527911 8146 raft_consensus.cc:374] T 0a5fdbfa16c946a38a90be15079bda0c P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.528012 8146 raft_consensus.cc:729] T 0a5fdbfa16c946a38a90be15079bda0c P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.527945 7194 tablet_service.cc:1467] Processing CreateTablet for tablet 357690713cd44eb3b2a297bfd6bbc17d (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=e508630eb48f45f88993a4b8e7fd2ea6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:04.528263 8146 consensus_queue.cc:260] T 0a5fdbfa16c946a38a90be15079bda0c P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:04.528401 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 357690713cd44eb3b2a297bfd6bbc17d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.528824 8146 ts_tablet_manager.cc:1398] T 0a5fdbfa16c946a38a90be15079bda0c P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:04.528925 8155 tablet_bootstrap.cc:654] T 0a5fdbfa16c946a38a90be15079bda0c P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.529522 8152 tablet_bootstrap.cc:492] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:04.529683 8152 ts_tablet_manager.cc:1367] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240502 21:27:04.530447 8152 raft_consensus.cc:348] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:04.530678 8152 raft_consensus.cc:374] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.530753 8152 raft_consensus.cc:729] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.531008 8152 consensus_queue.cc:260] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:04.531843 8146 tablet_bootstrap.cc:492] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:04.533512 8146 tablet_bootstrap.cc:654] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.534829 8152 ts_tablet_manager.cc:1398] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:27:04.538937 7970 raft_consensus.cc:482] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:04.539104 7970 raft_consensus.cc:504] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:04.539841 7970 leader_election.cc:288] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:04.540349 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c649fa0da35e4d47ba24f45a227f7e0b" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:27:04.540577 4440 raft_consensus.cc:2454] T c649fa0da35e4d47ba24f45a227f7e0b P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:04.541270 7148 leader_election.cc:302] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:04.541637 7970 raft_consensus.cc:2781] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:04.541738 7970 raft_consensus.cc:482] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:04.541842 7970 raft_consensus.cc:3037] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.542676 8152 tablet_bootstrap.cc:492] T 357690713cd44eb3b2a297bfd6bbc17d P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:04.545459 8154 tablet_bootstrap.cc:654] T 8f579ec914e540b28a1c7c2e42e498e8 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.545801 8152 tablet_bootstrap.cc:654] T 357690713cd44eb3b2a297bfd6bbc17d P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.548712 8154 tablet_bootstrap.cc:492] T 8f579ec914e540b28a1c7c2e42e498e8 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:04.548852 8154 ts_tablet_manager.cc:1367] T 8f579ec914e540b28a1c7c2e42e498e8 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20240502 21:27:04.549504 8154 raft_consensus.cc:348] T 8f579ec914e540b28a1c7c2e42e498e8 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:04.549666 8154 raft_consensus.cc:374] T 8f579ec914e540b28a1c7c2e42e498e8 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.549762 8154 raft_consensus.cc:729] T 8f579ec914e540b28a1c7c2e42e498e8 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.550007 8154 consensus_queue.cc:260] T 8f579ec914e540b28a1c7c2e42e498e8 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:04.550490 8154 ts_tablet_manager.cc:1398] T 8f579ec914e540b28a1c7c2e42e498e8 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:04.550729 8154 tablet_bootstrap.cc:492] T 357690713cd44eb3b2a297bfd6bbc17d P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:04.554040 8152 tablet_bootstrap.cc:492] T 357690713cd44eb3b2a297bfd6bbc17d P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:04.554188 8152 ts_tablet_manager.cc:1367] T 357690713cd44eb3b2a297bfd6bbc17d P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20240502 21:27:04.554970 8152 raft_consensus.cc:348] T 357690713cd44eb3b2a297bfd6bbc17d P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:04.555171 8152 raft_consensus.cc:374] T 357690713cd44eb3b2a297bfd6bbc17d P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.555286 8152 raft_consensus.cc:729] T 357690713cd44eb3b2a297bfd6bbc17d P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.555840 8152 consensus_queue.cc:260] T 357690713cd44eb3b2a297bfd6bbc17d P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:04.556358 8152 ts_tablet_manager.cc:1398] T 357690713cd44eb3b2a297bfd6bbc17d P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 21:27:04.558209 8154 tablet_bootstrap.cc:654] T 357690713cd44eb3b2a297bfd6bbc17d P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.560000 8154 tablet_bootstrap.cc:492] T 357690713cd44eb3b2a297bfd6bbc17d P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:04.560104 8146 tablet_bootstrap.cc:492] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:04.560267 8146 ts_tablet_manager.cc:1367] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.001s
I20240502 21:27:04.560271 8154 ts_tablet_manager.cc:1367] T 357690713cd44eb3b2a297bfd6bbc17d P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240502 21:27:04.561060 8154 raft_consensus.cc:348] T 357690713cd44eb3b2a297bfd6bbc17d P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:04.561151 8146 raft_consensus.cc:348] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:04.561287 8154 raft_consensus.cc:374] T 357690713cd44eb3b2a297bfd6bbc17d P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.561359 8154 raft_consensus.cc:729] T 357690713cd44eb3b2a297bfd6bbc17d P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.561371 8146 raft_consensus.cc:374] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.561473 8146 raft_consensus.cc:729] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.561623 8154 consensus_queue.cc:260] T 357690713cd44eb3b2a297bfd6bbc17d P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:04.561698 8146 consensus_queue.cc:260] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:04.562239 8154 ts_tablet_manager.cc:1398] T 357690713cd44eb3b2a297bfd6bbc17d P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:04.563040 7970 raft_consensus.cc:504] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:04.565819 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c649fa0da35e4d47ba24f45a227f7e0b" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:27:04.566105 4440 raft_consensus.cc:3037] T c649fa0da35e4d47ba24f45a227f7e0b P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.569233 4440 raft_consensus.cc:2454] T c649fa0da35e4d47ba24f45a227f7e0b P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:04.572211 8155 tablet_bootstrap.cc:492] T 0a5fdbfa16c946a38a90be15079bda0c P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:04.572176 7970 leader_election.cc:288] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:04.572417 8155 ts_tablet_manager.cc:1367] T 0a5fdbfa16c946a38a90be15079bda0c P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.045s user 0.009s sys 0.007s
I20240502 21:27:04.573449 8155 raft_consensus.cc:348] T 0a5fdbfa16c946a38a90be15079bda0c P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:04.573720 8155 raft_consensus.cc:374] T 0a5fdbfa16c946a38a90be15079bda0c P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.573835 8155 raft_consensus.cc:729] T 0a5fdbfa16c946a38a90be15079bda0c P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.574142 7148 leader_election.cc:302] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:04.574111 8155 consensus_queue.cc:260] T 0a5fdbfa16c946a38a90be15079bda0c P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:04.574579 7970 raft_consensus.cc:2781] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:04.574788 8155 ts_tablet_manager.cc:1398] T 0a5fdbfa16c946a38a90be15079bda0c P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 21:27:04.574803 7970 raft_consensus.cc:686] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:27:04.575491 7970 consensus_queue.cc:237] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:04.587095 8146 ts_tablet_manager.cc:1398] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.027s user 0.017s sys 0.004s
I20240502 21:27:04.587469 8146 tablet_bootstrap.cc:492] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:04.587877 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c649fa0da35e4d47ba24f45a227f7e0b" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:27:04.587970 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c649fa0da35e4d47ba24f45a227f7e0b" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:27:04.588171 4574 raft_consensus.cc:3037] T c649fa0da35e4d47ba24f45a227f7e0b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.589073 8146 tablet_bootstrap.cc:654] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.590565 4574 raft_consensus.cc:2454] T c649fa0da35e4d47ba24f45a227f7e0b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:04.590988 8146 tablet_bootstrap.cc:492] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:04.591163 8146 ts_tablet_manager.cc:1367] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:27:04.592110 8146 raft_consensus.cc:348] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:04.592296 8146 raft_consensus.cc:374] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.592382 8146 raft_consensus.cc:729] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.592615 8146 consensus_queue.cc:260] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:04.593024 8146 ts_tablet_manager.cc:1398] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:04.596068 4300 catalog_manager.cc:5484] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69). New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:04.612833 7970 raft_consensus.cc:482] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:04.612979 7970 raft_consensus.cc:504] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:04.613602 7970 leader_election.cc:288] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:04.614058 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f579ec914e540b28a1c7c2e42e498e8" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:04.614104 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f579ec914e540b28a1c7c2e42e498e8" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:04.614308 7618 raft_consensus.cc:2454] T 8f579ec914e540b28a1c7c2e42e498e8 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:04.614385 7350 raft_consensus.cc:2454] T 8f579ec914e540b28a1c7c2e42e498e8 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:04.614773 7149 leader_election.cc:302] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:04.615111 7970 raft_consensus.cc:2781] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:04.615237 7970 raft_consensus.cc:482] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:04.615325 7970 raft_consensus.cc:3037] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.617765 7970 raft_consensus.cc:504] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:04.618335 7970 leader_election.cc:288] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:04.618687 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f579ec914e540b28a1c7c2e42e498e8" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:04.618824 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f579ec914e540b28a1c7c2e42e498e8" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:04.618881 7350 raft_consensus.cc:3037] T 8f579ec914e540b28a1c7c2e42e498e8 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.618978 7618 raft_consensus.cc:3037] T 8f579ec914e540b28a1c7c2e42e498e8 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.620836 7350 raft_consensus.cc:2454] T 8f579ec914e540b28a1c7c2e42e498e8 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:04.620846 7618 raft_consensus.cc:2454] T 8f579ec914e540b28a1c7c2e42e498e8 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:04.621246 7149 leader_election.cc:302] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:04.621533 7970 raft_consensus.cc:2781] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:04.621716 7970 raft_consensus.cc:686] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:27:04.621987 7970 consensus_queue.cc:237] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:04.624760 4300 catalog_manager.cc:5484] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69). New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:04.709034 4300 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:04.714471 4574 raft_consensus.cc:1261] T c649fa0da35e4d47ba24f45a227f7e0b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:04.714653 4440 raft_consensus.cc:1261] T c649fa0da35e4d47ba24f45a227f7e0b P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:04.714996 7970 consensus_queue.cc:1035] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:04.715332 8163 consensus_queue.cc:1035] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:04.732472 8128 tablet.cc:1722] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:04.733023 8128 tablet.cc:1620] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:04.733819 4292 catalog_manager.cc:5576] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781) does not have the latest schema for tablet c649fa0da35e4d47ba24f45a227f7e0b (table rebalancer_dynamic_table [id=e0941c7363ad44e790631804380d445d]). Expected version 1 got 0
I20240502 21:27:04.741163 8174 tablet.cc:1722] T c649fa0da35e4d47ba24f45a227f7e0b P 7a736a8ebb164ce0a051c0fccd76c3b8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:04.743259 4292 catalog_manager.cc:5576] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781) does not have the latest schema for tablet c649fa0da35e4d47ba24f45a227f7e0b (table rebalancer_dynamic_table [id=e0941c7363ad44e790631804380d445d]). Expected version 1 got 0
I20240502 21:27:04.743440 8174 tablet.cc:1620] T c649fa0da35e4d47ba24f45a227f7e0b P 7a736a8ebb164ce0a051c0fccd76c3b8: MemRowSet was empty: no flush needed.
I20240502 21:27:04.744993 4300 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=e0941c7363ad44e790631804380d445d] alter complete (version 1)
I20240502 21:27:04.748165 8175 tablet.cc:1722] T c649fa0da35e4d47ba24f45a227f7e0b P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:04.750649 8175 tablet.cc:1620] T c649fa0da35e4d47ba24f45a227f7e0b P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:04.751310 4300 catalog_manager.cc:5576] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781) does not have the latest schema for tablet c649fa0da35e4d47ba24f45a227f7e0b (table rebalancer_dynamic_table [id=e0941c7363ad44e790631804380d445d]). Expected version 1 got 0
I20240502 21:27:04.753398 4292 catalog_manager.cc:5576] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781) does not have the latest schema for tablet c649fa0da35e4d47ba24f45a227f7e0b (table rebalancer_dynamic_table [id=e0941c7363ad44e790631804380d445d]). Expected version 1 got 0
I20240502 21:27:04.815630 8143 raft_consensus.cc:482] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:04.815848 8143 raft_consensus.cc:504] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:04.816514 8143 leader_election.cc:288] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:04.817047 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5fdbfa16c946a38a90be15079bda0c" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:04.817076 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5fdbfa16c946a38a90be15079bda0c" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:04.817314 7484 raft_consensus.cc:2454] T 0a5fdbfa16c946a38a90be15079bda0c P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 0.
I20240502 21:27:04.817332 7214 raft_consensus.cc:2454] T 0a5fdbfa16c946a38a90be15079bda0c P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 0.
I20240502 21:27:04.817809 7281 leader_election.cc:302] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:04.818300 8143 raft_consensus.cc:2781] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:04.818395 8143 raft_consensus.cc:482] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:04.818503 8143 raft_consensus.cc:3037] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.820731 8143 raft_consensus.cc:504] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:04.821341 8143 leader_election.cc:288] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Requested vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:04.821805 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5fdbfa16c946a38a90be15079bda0c" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:04.821827 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5fdbfa16c946a38a90be15079bda0c" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:04.822003 7214 raft_consensus.cc:3037] T 0a5fdbfa16c946a38a90be15079bda0c P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.822050 7484 raft_consensus.cc:3037] T 0a5fdbfa16c946a38a90be15079bda0c P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.824060 7214 raft_consensus.cc:2454] T 0a5fdbfa16c946a38a90be15079bda0c P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
I20240502 21:27:04.824472 7283 leader_election.cc:302] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:04.824721 8143 raft_consensus.cc:2781] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:04.824913 8143 raft_consensus.cc:686] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Becoming Leader. State: Replica: d0456c5e55a542128688904e003d403c, State: Running, Role: LEADER
I20240502 21:27:04.825178 8143 consensus_queue.cc:237] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:04.824905 7484 raft_consensus.cc:2454] T 0a5fdbfa16c946a38a90be15079bda0c P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
I20240502 21:27:04.828516 4292 catalog_manager.cc:5484] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c reported cstate change: term changed from 0 to 1, leader changed from <none> to d0456c5e55a542128688904e003d403c (127.29.82.70). New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:04.851716 4292 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:04.858856 8128 tablet.cc:1722] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:04.859403 8128 tablet.cc:1620] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:04.860136 8175 tablet.cc:1722] T c649fa0da35e4d47ba24f45a227f7e0b P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:04.860579 8175 tablet.cc:1620] T c649fa0da35e4d47ba24f45a227f7e0b P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:04.863330 8174 tablet.cc:1722] T c649fa0da35e4d47ba24f45a227f7e0b P 7a736a8ebb164ce0a051c0fccd76c3b8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:04.863817 8174 tablet.cc:1620] T c649fa0da35e4d47ba24f45a227f7e0b P 7a736a8ebb164ce0a051c0fccd76c3b8: MemRowSet was empty: no flush needed.
I20240502 21:27:04.870064 4292 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=e0941c7363ad44e790631804380d445d] alter complete (version 2)
I20240502 21:27:04.897081 7970 raft_consensus.cc:482] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:04.897275 7970 raft_consensus.cc:504] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:04.898025 7970 leader_election.cc:288] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:04.898334 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "357690713cd44eb3b2a297bfd6bbc17d" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:04.898576 7350 raft_consensus.cc:2454] T 357690713cd44eb3b2a297bfd6bbc17d P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:04.898556 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "357690713cd44eb3b2a297bfd6bbc17d" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:04.898828 7618 raft_consensus.cc:2454] T 357690713cd44eb3b2a297bfd6bbc17d P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:04.898990 7150 leader_election.cc:302] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:04.899267 7970 raft_consensus.cc:2781] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:04.899389 7970 raft_consensus.cc:482] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:04.899466 7970 raft_consensus.cc:3037] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.901798 7970 raft_consensus.cc:504] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:04.902292 7970 leader_election.cc:288] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:04.902745 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "357690713cd44eb3b2a297bfd6bbc17d" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:04.902789 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "357690713cd44eb3b2a297bfd6bbc17d" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:04.902915 7350 raft_consensus.cc:3037] T 357690713cd44eb3b2a297bfd6bbc17d P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.902969 7618 raft_consensus.cc:3037] T 357690713cd44eb3b2a297bfd6bbc17d P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.905119 7350 raft_consensus.cc:2454] T 357690713cd44eb3b2a297bfd6bbc17d P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:04.905486 7150 leader_election.cc:302] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:04.905527 7618 raft_consensus.cc:2454] T 357690713cd44eb3b2a297bfd6bbc17d P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:04.905838 7970 raft_consensus.cc:2781] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:04.905979 7970 raft_consensus.cc:686] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:27:04.906267 7970 consensus_queue.cc:237] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:04.909111 4300 catalog_manager.cc:5484] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69). New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:04.972404 4300 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:04.979925 8175 tablet.cc:1722] T c649fa0da35e4d47ba24f45a227f7e0b P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:04.980088 8174 tablet.cc:1722] T c649fa0da35e4d47ba24f45a227f7e0b P 7a736a8ebb164ce0a051c0fccd76c3b8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:04.979434 8128 tablet.cc:1722] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:04.989593 4298 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=e0941c7363ad44e790631804380d445d] alter complete (version 3)
I20240502 21:27:05.019143 7970 consensus_queue.cc:1035] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:05.023981 7970 consensus_queue.cc:1035] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:05.092087 4292 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:05.092316 4292 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:05.095248 4292 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet c649fa0da35e4d47ba24f45a227f7e0b
I20240502 21:27:05.096143 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet c649fa0da35e4d47ba24f45a227f7e0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:05 UTC) from {username='slave'} at 127.0.0.1:60018
I20240502 21:27:05.096333 7194 tablet_service.cc:1514] Processing DeleteTablet for tablet c649fa0da35e4d47ba24f45a227f7e0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:05 UTC) from {username='slave'} at 127.0.0.1:44652
I20240502 21:27:05.096724 8186 tablet_replica.cc:330] T c649fa0da35e4d47ba24f45a227f7e0b P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:27:05.096814 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet c649fa0da35e4d47ba24f45a227f7e0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:05 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:05.096954 8186 raft_consensus.cc:2229] T c649fa0da35e4d47ba24f45a227f7e0b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:05.097198 8186 raft_consensus.cc:2258] T c649fa0da35e4d47ba24f45a227f7e0b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:05.097411 8188 tablet_replica.cc:330] T c649fa0da35e4d47ba24f45a227f7e0b P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:05.097667 8188 raft_consensus.cc:2229] T c649fa0da35e4d47ba24f45a227f7e0b P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:05.097947 8188 raft_consensus.cc:2258] T c649fa0da35e4d47ba24f45a227f7e0b P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:05.098170 8186 ts_tablet_manager.cc:1875] T c649fa0da35e4d47ba24f45a227f7e0b P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:05.098537 8187 tablet_replica.cc:330] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:27:05.098779 8187 raft_consensus.cc:2229] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:05.098837 8188 ts_tablet_manager.cc:1875] T c649fa0da35e4d47ba24f45a227f7e0b P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:05.099164 8187 raft_consensus.cc:2258] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:05.099995 8187 ts_tablet_manager.cc:1875] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:05.104279 8186 ts_tablet_manager.cc:1888] T c649fa0da35e4d47ba24f45a227f7e0b P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:05.104439 8186 log.cc:1196] T c649fa0da35e4d47ba24f45a227f7e0b P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/c649fa0da35e4d47ba24f45a227f7e0b
I20240502 21:27:05.104864 8186 ts_tablet_manager.cc:1909] T c649fa0da35e4d47ba24f45a227f7e0b P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting consensus metadata
I20240502 21:27:05.105386 8188 ts_tablet_manager.cc:1888] T c649fa0da35e4d47ba24f45a227f7e0b P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:05.105512 8188 log.cc:1196] T c649fa0da35e4d47ba24f45a227f7e0b P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/c649fa0da35e4d47ba24f45a227f7e0b
I20240502 21:27:05.105809 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet c649fa0da35e4d47ba24f45a227f7e0b (table rebalancer_dynamic_table_new_name [id=e0941c7363ad44e790631804380d445d]) successfully deleted
I20240502 21:27:05.106037 8188 ts_tablet_manager.cc:1909] T c649fa0da35e4d47ba24f45a227f7e0b P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:05.106983 8187 ts_tablet_manager.cc:1888] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:05.107044 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet c649fa0da35e4d47ba24f45a227f7e0b (table rebalancer_dynamic_table_new_name [id=e0941c7363ad44e790631804380d445d]) successfully deleted
I20240502 21:27:05.107117 8187 log.cc:1196] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/c649fa0da35e4d47ba24f45a227f7e0b
I20240502 21:27:05.107463 8187 ts_tablet_manager.cc:1909] T c649fa0da35e4d47ba24f45a227f7e0b P 39981a8dfe94462b9a73ba489af4c714: Deleting consensus metadata
I20240502 21:27:05.108361 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet c649fa0da35e4d47ba24f45a227f7e0b (table rebalancer_dynamic_table_new_name [id=e0941c7363ad44e790631804380d445d]) successfully deleted
I20240502 21:27:05.198449 4292 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:05.205507 7464 tablet_service.cc:1467] Processing CreateTablet for tablet 9c5efc2e3fc04e60a3b5c80e8dd9887b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=165705447ce4483b8aeb0550e198908c]), partition=
I20240502 21:27:05.205562 4421 tablet_service.cc:1467] Processing CreateTablet for tablet 9c5efc2e3fc04e60a3b5c80e8dd9887b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=165705447ce4483b8aeb0550e198908c]), partition=
I20240502 21:27:05.205729 7194 tablet_service.cc:1467] Processing CreateTablet for tablet 9c5efc2e3fc04e60a3b5c80e8dd9887b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=165705447ce4483b8aeb0550e198908c]), partition=
I20240502 21:27:05.205965 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c5efc2e3fc04e60a3b5c80e8dd9887b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.205976 7464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c5efc2e3fc04e60a3b5c80e8dd9887b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.206081 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c5efc2e3fc04e60a3b5c80e8dd9887b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.212652 8190 tablet_bootstrap.cc:492] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:05.214447 8190 tablet_bootstrap.cc:654] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.219611 8192 tablet_bootstrap.cc:492] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:05.221993 8192 tablet_bootstrap.cc:654] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.224293 8191 tablet_bootstrap.cc:492] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:05.226502 8191 tablet_bootstrap.cc:654] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.232826 8192 tablet_bootstrap.cc:492] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:05.233006 8192 ts_tablet_manager.cc:1367] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.004s
I20240502 21:27:05.234524 8192 raft_consensus.cc:348] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:05.234812 8192 raft_consensus.cc:374] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.234895 8192 raft_consensus.cc:729] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.235144 8192 consensus_queue.cc:260] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:05.235913 8192 ts_tablet_manager.cc:1398] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240502 21:27:05.237116 8191 tablet_bootstrap.cc:492] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:05.237273 8191 ts_tablet_manager.cc:1367] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20240502 21:27:05.237896 8191 raft_consensus.cc:348] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:05.238085 8191 raft_consensus.cc:374] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.238180 8191 raft_consensus.cc:729] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.238473 8191 consensus_queue.cc:260] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:05.239979 8191 ts_tablet_manager.cc:1398] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:27:05.242117 8190 tablet_bootstrap.cc:492] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:05.242456 8190 ts_tablet_manager.cc:1367] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.016s
I20240502 21:27:05.243330 8190 raft_consensus.cc:348] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:05.243554 8190 raft_consensus.cc:374] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.243680 8190 raft_consensus.cc:729] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.243898 8190 consensus_queue.cc:260] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:05.244444 8190 ts_tablet_manager.cc:1398] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:27:05.253880 7970 raft_consensus.cc:482] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:05.254025 7970 raft_consensus.cc:504] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:05.254590 7970 leader_election.cc:288] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:05.255024 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5efc2e3fc04e60a3b5c80e8dd9887b" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:05.255138 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5efc2e3fc04e60a3b5c80e8dd9887b" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:27:05.255239 7484 raft_consensus.cc:2454] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:05.255360 4440 raft_consensus.cc:2454] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:05.255705 7148 leader_election.cc:302] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:05.255947 7970 raft_consensus.cc:2781] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:05.256031 7970 raft_consensus.cc:482] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:05.256143 7970 raft_consensus.cc:3037] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.258193 7970 raft_consensus.cc:504] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:05.258714 7970 leader_election.cc:288] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:05.259100 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5efc2e3fc04e60a3b5c80e8dd9887b" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:05.259169 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5efc2e3fc04e60a3b5c80e8dd9887b" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:27:05.259268 7484 raft_consensus.cc:3037] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.259358 4440 raft_consensus.cc:3037] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.261428 7484 raft_consensus.cc:2454] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:05.261538 4440 raft_consensus.cc:2454] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:05.261819 7148 leader_election.cc:302] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:05.262120 7970 raft_consensus.cc:2781] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:05.262310 7970 raft_consensus.cc:686] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:27:05.262552 7970 consensus_queue.cc:237] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:05.265026 4292 catalog_manager.cc:5484] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69). New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:05.305338 8143 consensus_queue.cc:1035] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:05.316498 8143 consensus_queue.cc:1035] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:05.322136 8177 consensus_queue.cc:1035] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:05.329380 8177 consensus_queue.cc:1035] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:05.371762 4292 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:05.377637 4440 raft_consensus.cc:1261] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:05.377612 7484 raft_consensus.cc:1261] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:05.378314 8177 consensus_queue.cc:1035] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:05.378641 7970 consensus_queue.cc:1035] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:05.382755 4292 catalog_manager.cc:5576] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781) does not have the latest schema for tablet 9c5efc2e3fc04e60a3b5c80e8dd9887b (table rebalancer_dynamic_table [id=165705447ce4483b8aeb0550e198908c]). Expected version 1 got 0
I20240502 21:27:05.383302 4300 catalog_manager.cc:5576] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913) does not have the latest schema for tablet 9c5efc2e3fc04e60a3b5c80e8dd9887b (table rebalancer_dynamic_table [id=165705447ce4483b8aeb0550e198908c]). Expected version 1 got 0
I20240502 21:27:05.383637 8128 tablet.cc:1722] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:05.384160 8128 tablet.cc:1620] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:05.385483 8175 tablet.cc:1722] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:05.386009 8175 tablet.cc:1620] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:05.390472 8204 tablet.cc:1722] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:05.391229 8204 tablet.cc:1620] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:05.393563 4291 catalog_manager.cc:5576] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913) does not have the latest schema for tablet 9c5efc2e3fc04e60a3b5c80e8dd9887b (table rebalancer_dynamic_table [id=165705447ce4483b8aeb0550e198908c]). Expected version 1 got 0
I20240502 21:27:05.395778 4300 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=165705447ce4483b8aeb0550e198908c] alter complete (version 1)
I20240502 21:27:05.492767 4300 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:05.499402 4300 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:05.501387 4818 tablet_service.cc:1467] Processing CreateTablet for tablet a417562f482e4d468095fbe833056233 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=e7b935c5901749939ea064dc2f41c619]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:05.501863 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a417562f482e4d468095fbe833056233. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.502662 7595 tablet_service.cc:1467] Processing CreateTablet for tablet a417562f482e4d468095fbe833056233 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=e7b935c5901749939ea064dc2f41c619]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:05.502749 7596 tablet_service.cc:1467] Processing CreateTablet for tablet 2ae670e3d8ae49f2b8c7df65b45b179e (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=e7b935c5901749939ea064dc2f41c619]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:05.503134 7595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a417562f482e4d468095fbe833056233. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.503346 7596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ae670e3d8ae49f2b8c7df65b45b179e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.504746 7330 tablet_service.cc:1467] Processing CreateTablet for tablet 2ae670e3d8ae49f2b8c7df65b45b179e (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=e7b935c5901749939ea064dc2f41c619]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:05.504796 7328 tablet_service.cc:1467] Processing CreateTablet for tablet 39260f3652b9473687ce1abb9c39223e (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=e7b935c5901749939ea064dc2f41c619]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:05.505189 7330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ae670e3d8ae49f2b8c7df65b45b179e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.505506 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39260f3652b9473687ce1abb9c39223e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.509975 7194 tablet_service.cc:1467] Processing CreateTablet for tablet a417562f482e4d468095fbe833056233 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=e7b935c5901749939ea064dc2f41c619]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:05.510442 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a417562f482e4d468095fbe833056233. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.511471 8205 tablet_bootstrap.cc:492] T 2ae670e3d8ae49f2b8c7df65b45b179e P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:05.513331 8205 tablet_bootstrap.cc:654] T 2ae670e3d8ae49f2b8c7df65b45b179e P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.518433 8191 tablet_bootstrap.cc:492] T a417562f482e4d468095fbe833056233 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:05.519723 7462 tablet_service.cc:1467] Processing CreateTablet for tablet 39260f3652b9473687ce1abb9c39223e (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=e7b935c5901749939ea064dc2f41c619]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:05.520071 8191 tablet_bootstrap.cc:654] T a417562f482e4d468095fbe833056233 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.520165 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39260f3652b9473687ce1abb9c39223e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.519716 7464 tablet_service.cc:1467] Processing CreateTablet for tablet 2ae670e3d8ae49f2b8c7df65b45b179e (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=e7b935c5901749939ea064dc2f41c619]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:05.521287 7464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ae670e3d8ae49f2b8c7df65b45b179e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.524231 4421 tablet_service.cc:1467] Processing CreateTablet for tablet 39260f3652b9473687ce1abb9c39223e (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=e7b935c5901749939ea064dc2f41c619]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:05.524694 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39260f3652b9473687ce1abb9c39223e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.527560 8190 tablet_bootstrap.cc:492] T 39260f3652b9473687ce1abb9c39223e P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:05.529249 8190 tablet_bootstrap.cc:654] T 39260f3652b9473687ce1abb9c39223e P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.533298 8192 tablet_bootstrap.cc:492] T 39260f3652b9473687ce1abb9c39223e P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:05.535447 8192 tablet_bootstrap.cc:654] T 39260f3652b9473687ce1abb9c39223e P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.538746 8128 tablet.cc:1722] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:05.539245 8128 tablet.cc:1620] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:05.539633 8190 tablet_bootstrap.cc:492] T 39260f3652b9473687ce1abb9c39223e P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:05.539695 8204 tablet.cc:1722] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:05.539796 8190 ts_tablet_manager.cc:1367] T 39260f3652b9473687ce1abb9c39223e P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20240502 21:27:05.540104 8204 tablet.cc:1620] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:05.541003 8190 raft_consensus.cc:348] T 39260f3652b9473687ce1abb9c39223e P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:05.541250 8190 raft_consensus.cc:374] T 39260f3652b9473687ce1abb9c39223e P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.541323 8190 raft_consensus.cc:729] T 39260f3652b9473687ce1abb9c39223e P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.542086 8190 consensus_queue.cc:260] T 39260f3652b9473687ce1abb9c39223e P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:05.545197 8190 ts_tablet_manager.cc:1398] T 39260f3652b9473687ce1abb9c39223e P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:27:05.545409 8175 tablet.cc:1722] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:05.545542 8190 tablet_bootstrap.cc:492] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:05.545904 8175 tablet.cc:1620] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:05.547701 8190 tablet_bootstrap.cc:654] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.549134 8207 tablet_bootstrap.cc:492] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:05.551926 8192 tablet_bootstrap.cc:492] T 39260f3652b9473687ce1abb9c39223e P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:05.552124 8192 ts_tablet_manager.cc:1367] T 39260f3652b9473687ce1abb9c39223e P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20240502 21:27:05.552637 8205 tablet_bootstrap.cc:492] T 2ae670e3d8ae49f2b8c7df65b45b179e P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:05.552843 8205 ts_tablet_manager.cc:1367] T 2ae670e3d8ae49f2b8c7df65b45b179e P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.000s
I20240502 21:27:05.553733 4291 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=165705447ce4483b8aeb0550e198908c] alter complete (version 2)
I20240502 21:27:05.554270 8205 raft_consensus.cc:348] T 2ae670e3d8ae49f2b8c7df65b45b179e P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:05.555189 8205 raft_consensus.cc:374] T 2ae670e3d8ae49f2b8c7df65b45b179e P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.555466 8205 raft_consensus.cc:729] T 2ae670e3d8ae49f2b8c7df65b45b179e P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.555959 8210 tablet_bootstrap.cc:492] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:05.555924 8205 consensus_queue.cc:260] T 2ae670e3d8ae49f2b8c7df65b45b179e P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:05.557181 8205 ts_tablet_manager.cc:1398] T 2ae670e3d8ae49f2b8c7df65b45b179e P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:05.558140 8210 tablet_bootstrap.cc:654] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.558415 8205 tablet_bootstrap.cc:492] T 39260f3652b9473687ce1abb9c39223e P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:05.560266 8205 tablet_bootstrap.cc:654] T 39260f3652b9473687ce1abb9c39223e P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.561399 8192 raft_consensus.cc:348] T 39260f3652b9473687ce1abb9c39223e P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:05.561622 8192 raft_consensus.cc:374] T 39260f3652b9473687ce1abb9c39223e P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.561841 8192 raft_consensus.cc:729] T 39260f3652b9473687ce1abb9c39223e P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.562428 8192 consensus_queue.cc:260] T 39260f3652b9473687ce1abb9c39223e P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:05.563967 8192 ts_tablet_manager.cc:1398] T 39260f3652b9473687ce1abb9c39223e P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20240502 21:27:05.564311 8207 tablet_bootstrap.cc:654] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.564549 8210 tablet_bootstrap.cc:492] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:05.564746 8210 ts_tablet_manager.cc:1367] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20240502 21:27:05.565125 8072 raft_consensus.cc:482] T 39260f3652b9473687ce1abb9c39223e P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:05.565258 8072 raft_consensus.cc:504] T 39260f3652b9473687ce1abb9c39223e P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:05.565536 8210 raft_consensus.cc:348] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:05.565671 8210 raft_consensus.cc:374] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.565722 8210 raft_consensus.cc:729] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.565992 8072 leader_election.cc:288] T 39260f3652b9473687ce1abb9c39223e P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:05.565903 8210 consensus_queue.cc:260] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:05.566448 8210 ts_tablet_manager.cc:1398] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:05.566567 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39260f3652b9473687ce1abb9c39223e" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:27:05.566748 4440 raft_consensus.cc:2454] T 39260f3652b9473687ce1abb9c39223e P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:05.567237 7416 leader_election.cc:302] T 39260f3652b9473687ce1abb9c39223e P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:05.567493 8072 raft_consensus.cc:2781] T 39260f3652b9473687ce1abb9c39223e P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:05.567656 8072 raft_consensus.cc:482] T 39260f3652b9473687ce1abb9c39223e P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:05.567750 8072 raft_consensus.cc:3037] T 39260f3652b9473687ce1abb9c39223e P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.567746 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39260f3652b9473687ce1abb9c39223e" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
W20240502 21:27:05.568922 7419 leader_election.cc:341] T 39260f3652b9473687ce1abb9c39223e P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:05.569851 8207 tablet_bootstrap.cc:492] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:05.570070 8207 ts_tablet_manager.cc:1367] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240502 21:27:05.570376 8072 raft_consensus.cc:504] T 39260f3652b9473687ce1abb9c39223e P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:05.571512 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39260f3652b9473687ce1abb9c39223e" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:27:05.571722 4440 raft_consensus.cc:3037] T 39260f3652b9473687ce1abb9c39223e P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.571805 8191 tablet_bootstrap.cc:492] T a417562f482e4d468095fbe833056233 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:05.571969 8191 ts_tablet_manager.cc:1367] T a417562f482e4d468095fbe833056233 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.054s user 0.005s sys 0.005s
I20240502 21:27:05.572440 8207 raft_consensus.cc:348] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:05.572712 8207 raft_consensus.cc:374] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.572844 8207 raft_consensus.cc:729] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.572733 8191 raft_consensus.cc:348] T a417562f482e4d468095fbe833056233 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:05.572954 8191 raft_consensus.cc:374] T a417562f482e4d468095fbe833056233 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.573050 8191 raft_consensus.cc:729] T a417562f482e4d468095fbe833056233 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.573232 8207 consensus_queue.cc:260] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:05.573321 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39260f3652b9473687ce1abb9c39223e" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:05.573825 8207 ts_tablet_manager.cc:1398] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20240502 21:27:05.573809 7419 leader_election.cc:341] T 39260f3652b9473687ce1abb9c39223e P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:05.573987 8072 leader_election.cc:288] T 39260f3652b9473687ce1abb9c39223e P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:05.574206 8207 tablet_bootstrap.cc:492] T a417562f482e4d468095fbe833056233 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:05.574445 4440 raft_consensus.cc:2454] T 39260f3652b9473687ce1abb9c39223e P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:05.574906 7416 leader_election.cc:302] T 39260f3652b9473687ce1abb9c39223e P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 71e1375b799b439fa3c83ca763c039c1; no voters: d0456c5e55a542128688904e003d403c
I20240502 21:27:05.574942 8191 consensus_queue.cc:260] T a417562f482e4d468095fbe833056233 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:05.575472 8072 raft_consensus.cc:2781] T 39260f3652b9473687ce1abb9c39223e P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:05.577066 8191 ts_tablet_manager.cc:1398] T a417562f482e4d468095fbe833056233 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:27:05.578612 8207 tablet_bootstrap.cc:654] T a417562f482e4d468095fbe833056233 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.580045 8190 tablet_bootstrap.cc:492] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:05.580186 8190 ts_tablet_manager.cc:1367] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.002s
I20240502 21:27:05.581884 8190 raft_consensus.cc:348] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:05.582017 8205 tablet_bootstrap.cc:492] T 39260f3652b9473687ce1abb9c39223e P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:05.582089 8190 raft_consensus.cc:374] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.582182 8190 raft_consensus.cc:729] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.582177 8205 ts_tablet_manager.cc:1367] T 39260f3652b9473687ce1abb9c39223e P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
I20240502 21:27:05.583539 8205 raft_consensus.cc:348] T 39260f3652b9473687ce1abb9c39223e P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:05.583834 8205 raft_consensus.cc:374] T 39260f3652b9473687ce1abb9c39223e P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.583539 8190 consensus_queue.cc:260] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:05.583914 8205 raft_consensus.cc:729] T 39260f3652b9473687ce1abb9c39223e P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.584205 8205 consensus_queue.cc:260] T 39260f3652b9473687ce1abb9c39223e P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:05.584805 8190 ts_tablet_manager.cc:1398] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:27:05.587505 8205 ts_tablet_manager.cc:1398] T 39260f3652b9473687ce1abb9c39223e P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 21:27:05.587864 8072 raft_consensus.cc:686] T 39260f3652b9473687ce1abb9c39223e P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Becoming Leader. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Running, Role: LEADER
I20240502 21:27:05.588212 8072 consensus_queue.cc:237] T 39260f3652b9473687ce1abb9c39223e P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:05.592334 4292 catalog_manager.cc:5484] T 39260f3652b9473687ce1abb9c39223e P 31a72286397c4edc9a63eca3412be168 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a72286397c4edc9a63eca3412be168 (127.29.82.71). New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:05.608075 8207 tablet_bootstrap.cc:492] T a417562f482e4d468095fbe833056233 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:05.608276 8207 ts_tablet_manager.cc:1367] T a417562f482e4d468095fbe833056233 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.002s
I20240502 21:27:05.608911 8207 raft_consensus.cc:348] T a417562f482e4d468095fbe833056233 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:05.609109 8207 raft_consensus.cc:374] T a417562f482e4d468095fbe833056233 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.609194 8207 raft_consensus.cc:729] T a417562f482e4d468095fbe833056233 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.609418 8207 consensus_queue.cc:260] T a417562f482e4d468095fbe833056233 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:05.609967 8207 ts_tablet_manager.cc:1398] T a417562f482e4d468095fbe833056233 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:05.656250 4292 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:05.663877 8128 tablet.cc:1722] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:05.664650 8204 tablet.cc:1722] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:05.665253 8175 tablet.cc:1722] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:05.672606 4292 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=165705447ce4483b8aeb0550e198908c] alter complete (version 3)
I20240502 21:27:05.675755 8072 raft_consensus.cc:482] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:05.675930 8072 raft_consensus.cc:504] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:05.676563 8072 leader_election.cc:288] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:05.677021 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae670e3d8ae49f2b8c7df65b45b179e" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:05.677316 7618 raft_consensus.cc:2454] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:05.677249 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae670e3d8ae49f2b8c7df65b45b179e" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:05.677495 7349 raft_consensus.cc:2454] T 2ae670e3d8ae49f2b8c7df65b45b179e P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:05.677731 7418 leader_election.cc:302] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168; no voters:
I20240502 21:27:05.678030 8072 raft_consensus.cc:2781] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:05.678120 8072 raft_consensus.cc:482] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:05.678225 8072 raft_consensus.cc:3037] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.680131 8072 raft_consensus.cc:504] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:05.680687 8072 leader_election.cc:288] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Requested vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:05.681082 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae670e3d8ae49f2b8c7df65b45b179e" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:05.681142 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae670e3d8ae49f2b8c7df65b45b179e" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:05.681247 7618 raft_consensus.cc:3037] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.681358 7349 raft_consensus.cc:3037] T 2ae670e3d8ae49f2b8c7df65b45b179e P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.684043 7618 raft_consensus.cc:2454] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:05.684100 7349 raft_consensus.cc:2454] T 2ae670e3d8ae49f2b8c7df65b45b179e P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:05.684454 7418 leader_election.cc:302] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168; no voters:
I20240502 21:27:05.684718 8072 raft_consensus.cc:2781] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:05.684872 8072 raft_consensus.cc:686] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Becoming Leader. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Running, Role: LEADER
I20240502 21:27:05.685129 8072 consensus_queue.cc:237] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:05.687994 4300 catalog_manager.cc:5484] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a72286397c4edc9a63eca3412be168 (127.29.82.71). New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:05.750988 8024 raft_consensus.cc:482] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:05.751183 8024 raft_consensus.cc:504] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:05.751965 8024 leader_election.cc:288] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:05.752468 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a417562f482e4d468095fbe833056233" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:05.752719 7215 raft_consensus.cc:2454] T a417562f482e4d468095fbe833056233 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:27:05.753165 4772 leader_election.cc:302] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:05.753428 8024 raft_consensus.cc:2781] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:05.753525 8024 raft_consensus.cc:482] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:05.753640 8024 raft_consensus.cc:3037] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.756243 8024 raft_consensus.cc:504] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:05.756940 8024 leader_election.cc:288] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:05.760332 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a417562f482e4d468095fbe833056233" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:05.760560 7215 raft_consensus.cc:3037] T a417562f482e4d468095fbe833056233 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.763345 7215 raft_consensus.cc:2454] T a417562f482e4d468095fbe833056233 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:27:05.763833 4772 leader_election.cc:302] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:05.764181 8024 raft_consensus.cc:2781] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:05.764366 8024 raft_consensus.cc:686] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:27:05.764737 8024 consensus_queue.cc:237] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:05.771459 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a417562f482e4d468095fbe833056233" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:05.771607 7617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a417562f482e4d468095fbe833056233" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:05.771791 7618 raft_consensus.cc:2454] T a417562f482e4d468095fbe833056233 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:27:05.778080 4300 catalog_manager.cc:5484] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:05.780328 4291 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:05.780655 4291 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:05.790237 4291 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 9c5efc2e3fc04e60a3b5c80e8dd9887b
I20240502 21:27:05.792142 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c5efc2e3fc04e60a3b5c80e8dd9887b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:05 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:05.792335 7194 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c5efc2e3fc04e60a3b5c80e8dd9887b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:05 UTC) from {username='slave'} at 127.0.0.1:44652
I20240502 21:27:05.792310 7464 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c5efc2e3fc04e60a3b5c80e8dd9887b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:05 UTC) from {username='slave'} at 127.0.0.1:37182
I20240502 21:27:05.811780 8226 tablet_replica.cc:330] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:27:05.812603 8226 raft_consensus.cc:2229] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:05.813666 8226 raft_consensus.cc:2258] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:05.820855 8226 ts_tablet_manager.cc:1875] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:05.826825 8227 tablet_replica.cc:330] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:27:05.830814 8227 raft_consensus.cc:2229] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:05.833909 8227 raft_consensus.cc:2258] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:05.837241 8227 ts_tablet_manager.cc:1875] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:05.840523 8225 tablet_replica.cc:330] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:05.841194 8225 raft_consensus.cc:2229] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:05.841869 8225 raft_consensus.cc:2258] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:05.846737 8225 ts_tablet_manager.cc:1875] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:05.857637 8226 ts_tablet_manager.cc:1888] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:05.857924 8226 log.cc:1196] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/9c5efc2e3fc04e60a3b5c80e8dd9887b
I20240502 21:27:05.858835 8226 ts_tablet_manager.cc:1909] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 39981a8dfe94462b9a73ba489af4c714: Deleting consensus metadata
I20240502 21:27:05.860982 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet 9c5efc2e3fc04e60a3b5c80e8dd9887b (table rebalancer_dynamic_table_new_name [id=165705447ce4483b8aeb0550e198908c]) successfully deleted
I20240502 21:27:05.861511 8227 ts_tablet_manager.cc:1888] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:05.861846 8227 log.cc:1196] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/9c5efc2e3fc04e60a3b5c80e8dd9887b
I20240502 21:27:05.864365 8227 ts_tablet_manager.cc:1909] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 31a72286397c4edc9a63eca3412be168: Deleting consensus metadata
I20240502 21:27:05.866379 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet 9c5efc2e3fc04e60a3b5c80e8dd9887b (table rebalancer_dynamic_table_new_name [id=165705447ce4483b8aeb0550e198908c]) successfully deleted
I20240502 21:27:05.873524 8225 ts_tablet_manager.cc:1888] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:05.873775 8225 log.cc:1196] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/9c5efc2e3fc04e60a3b5c80e8dd9887b
I20240502 21:27:05.874714 8225 ts_tablet_manager.cc:1909] T 9c5efc2e3fc04e60a3b5c80e8dd9887b P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:05.876574 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet 9c5efc2e3fc04e60a3b5c80e8dd9887b (table rebalancer_dynamic_table_new_name [id=165705447ce4483b8aeb0550e198908c]) successfully deleted
I20240502 21:27:05.895772 4291 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:05.909826 7464 tablet_service.cc:1467] Processing CreateTablet for tablet e0e436a62aab4f8ea5851d7c5d821fee (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a4a396e57c1f40dc8d7cd7d2935758e9]), partition=
I20240502 21:27:05.910717 7464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0e436a62aab4f8ea5851d7c5d821fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.910674 4421 tablet_service.cc:1467] Processing CreateTablet for tablet e0e436a62aab4f8ea5851d7c5d821fee (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a4a396e57c1f40dc8d7cd7d2935758e9]), partition=
I20240502 21:27:05.911504 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0e436a62aab4f8ea5851d7c5d821fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.912900 7328 tablet_service.cc:1467] Processing CreateTablet for tablet e0e436a62aab4f8ea5851d7c5d821fee (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a4a396e57c1f40dc8d7cd7d2935758e9]), partition=
I20240502 21:27:05.913585 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0e436a62aab4f8ea5851d7c5d821fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.924882 8190 tablet_bootstrap.cc:492] T e0e436a62aab4f8ea5851d7c5d821fee P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:05.925068 8192 tablet_bootstrap.cc:492] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:05.928629 8190 tablet_bootstrap.cc:654] T e0e436a62aab4f8ea5851d7c5d821fee P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.928707 8192 tablet_bootstrap.cc:654] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.929929 8205 tablet_bootstrap.cc:492] T e0e436a62aab4f8ea5851d7c5d821fee P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:05.933279 8190 tablet_bootstrap.cc:492] T e0e436a62aab4f8ea5851d7c5d821fee P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:05.933601 8190 ts_tablet_manager.cc:1367] T e0e436a62aab4f8ea5851d7c5d821fee P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240502 21:27:05.934090 8205 tablet_bootstrap.cc:654] T e0e436a62aab4f8ea5851d7c5d821fee P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.936004 8190 raft_consensus.cc:348] T e0e436a62aab4f8ea5851d7c5d821fee P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:05.936599 8190 raft_consensus.cc:374] T e0e436a62aab4f8ea5851d7c5d821fee P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.936767 8190 raft_consensus.cc:729] T e0e436a62aab4f8ea5851d7c5d821fee P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.937652 8190 consensus_queue.cc:260] T e0e436a62aab4f8ea5851d7c5d821fee P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:05.938951 8190 ts_tablet_manager.cc:1398] T e0e436a62aab4f8ea5851d7c5d821fee P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 21:27:05.941653 8205 tablet_bootstrap.cc:492] T e0e436a62aab4f8ea5851d7c5d821fee P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:05.941973 8205 ts_tablet_manager.cc:1367] T e0e436a62aab4f8ea5851d7c5d821fee P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20240502 21:27:05.943377 8205 raft_consensus.cc:348] T e0e436a62aab4f8ea5851d7c5d821fee P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:05.943730 8205 raft_consensus.cc:374] T e0e436a62aab4f8ea5851d7c5d821fee P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.943941 8205 raft_consensus.cc:729] T e0e436a62aab4f8ea5851d7c5d821fee P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.944454 8205 consensus_queue.cc:260] T e0e436a62aab4f8ea5851d7c5d821fee P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:05.945724 8192 tablet_bootstrap.cc:492] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:05.946055 8192 ts_tablet_manager.cc:1367] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.003s
I20240502 21:27:05.946504 8205 ts_tablet_manager.cc:1398] T e0e436a62aab4f8ea5851d7c5d821fee P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.004s user 0.003s sys 0.002s
I20240502 21:27:05.948164 8192 raft_consensus.cc:348] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:05.948664 8192 raft_consensus.cc:374] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.948866 8192 raft_consensus.cc:729] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.949335 8192 consensus_queue.cc:260] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:05.950529 8192 ts_tablet_manager.cc:1398] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 21:27:05.970561 8074 raft_consensus.cc:482] T 39260f3652b9473687ce1abb9c39223e P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:05.971046 8074 raft_consensus.cc:504] T 39260f3652b9473687ce1abb9c39223e P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:05.973183 8074 leader_election.cc:288] T 39260f3652b9473687ce1abb9c39223e P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:05.974586 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39260f3652b9473687ce1abb9c39223e" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:05.993472 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39260f3652b9473687ce1abb9c39223e" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:27:05.994153 4440 raft_consensus.cc:2379] T 39260f3652b9473687ce1abb9c39223e P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0456c5e55a542128688904e003d403c in current term 1: Already voted for candidate 31a72286397c4edc9a63eca3412be168 in this term.
I20240502 21:27:05.995003 7281 leader_election.cc:302] T 39260f3652b9473687ce1abb9c39223e P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0456c5e55a542128688904e003d403c; no voters: 31a72286397c4edc9a63eca3412be168, 71e1375b799b439fa3c83ca763c039c1
I20240502 21:27:05.995497 8074 raft_consensus.cc:3037] T 39260f3652b9473687ce1abb9c39223e P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.006004 8074 raft_consensus.cc:2726] T 39260f3652b9473687ce1abb9c39223e P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:27:06.057777 8072 consensus_queue.cc:1035] T 39260f3652b9473687ce1abb9c39223e P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:06.066630 8215 consensus_queue.cc:1035] T 39260f3652b9473687ce1abb9c39223e P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:06.080397 8072 consensus_queue.cc:1035] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:06.101377 8072 consensus_queue.cc:1035] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:06.163678 8168 raft_consensus.cc:482] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:06.163856 8168 raft_consensus.cc:504] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:06.164702 8168 leader_election.cc:288] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:06.165671 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e436a62aab4f8ea5851d7c5d821fee" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:06.165939 7349 raft_consensus.cc:2454] T e0e436a62aab4f8ea5851d7c5d821fee P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:06.166369 4377 leader_election.cc:302] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:06.166746 8168 raft_consensus.cc:2781] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:06.166869 8168 raft_consensus.cc:482] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:06.166960 8168 raft_consensus.cc:3037] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.169739 8168 raft_consensus.cc:504] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:06.170358 8168 leader_election.cc:288] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:06.170835 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e436a62aab4f8ea5851d7c5d821fee" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:06.171051 7349 raft_consensus.cc:3037] T e0e436a62aab4f8ea5851d7c5d821fee P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.173168 7349 raft_consensus.cc:2454] T e0e436a62aab4f8ea5851d7c5d821fee P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:06.173452 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e436a62aab4f8ea5851d7c5d821fee" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:06.173640 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e436a62aab4f8ea5851d7c5d821fee" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:06.173679 4377 leader_election.cc:302] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:06.173702 7484 raft_consensus.cc:2454] T e0e436a62aab4f8ea5851d7c5d821fee P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:06.174593 8168 raft_consensus.cc:2781] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:06.174747 8168 raft_consensus.cc:686] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:27:06.175035 8168 consensus_queue.cc:237] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:06.178397 4300 catalog_manager.cc:5484] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:06.216610 8120 consensus_queue.cc:1035] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:06.221719 7617 raft_consensus.cc:3037] T a417562f482e4d468095fbe833056233 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.225373 8120 consensus_queue.cc:1035] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:06.292886 4291 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:06.297360 4300 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:06.303809 4818 tablet_service.cc:1467] Processing CreateTablet for tablet acdc893a70224a7b82083e998910d6ef (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=23c1b78fb75948c193bd4eb1c32f8bef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:06.304366 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acdc893a70224a7b82083e998910d6ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.304608 4817 tablet_service.cc:1467] Processing CreateTablet for tablet 686a74b8c1034112b293c4c7c6bcd99a (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=23c1b78fb75948c193bd4eb1c32f8bef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:06.304929 7596 tablet_service.cc:1467] Processing CreateTablet for tablet 9a2616aa29b74980920210ac546c464b (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=23c1b78fb75948c193bd4eb1c32f8bef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:06.305109 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 686a74b8c1034112b293c4c7c6bcd99a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.305370 7596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a2616aa29b74980920210ac546c464b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.308693 7464 tablet_service.cc:1467] Processing CreateTablet for tablet acdc893a70224a7b82083e998910d6ef (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=23c1b78fb75948c193bd4eb1c32f8bef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:06.309144 7464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acdc893a70224a7b82083e998910d6ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.311939 7462 tablet_service.cc:1467] Processing CreateTablet for tablet 686a74b8c1034112b293c4c7c6bcd99a (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=23c1b78fb75948c193bd4eb1c32f8bef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:06.312413 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 686a74b8c1034112b293c4c7c6bcd99a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.316828 7330 tablet_service.cc:1467] Processing CreateTablet for tablet 686a74b8c1034112b293c4c7c6bcd99a (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=23c1b78fb75948c193bd4eb1c32f8bef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:06.317317 7330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 686a74b8c1034112b293c4c7c6bcd99a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.316828 7328 tablet_service.cc:1467] Processing CreateTablet for tablet acdc893a70224a7b82083e998910d6ef (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=23c1b78fb75948c193bd4eb1c32f8bef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:06.319990 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acdc893a70224a7b82083e998910d6ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.322870 8251 tablet_bootstrap.cc:492] T 9a2616aa29b74980920210ac546c464b P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:06.322893 8190 tablet_bootstrap.cc:492] T 686a74b8c1034112b293c4c7c6bcd99a P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:06.324657 8190 tablet_bootstrap.cc:654] T 686a74b8c1034112b293c4c7c6bcd99a P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.328966 7329 tablet_service.cc:1467] Processing CreateTablet for tablet 9a2616aa29b74980920210ac546c464b (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=23c1b78fb75948c193bd4eb1c32f8bef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:06.329394 7329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a2616aa29b74980920210ac546c464b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.329515 7462 tablet_service.cc:1467] Processing CreateTablet for tablet 9a2616aa29b74980920210ac546c464b (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=23c1b78fb75948c193bd4eb1c32f8bef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:06.329656 8190 tablet_bootstrap.cc:492] T 686a74b8c1034112b293c4c7c6bcd99a P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:06.329816 8190 ts_tablet_manager.cc:1367] T 686a74b8c1034112b293c4c7c6bcd99a P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 21:27:06.329947 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a2616aa29b74980920210ac546c464b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.329953 8251 tablet_bootstrap.cc:654] T 9a2616aa29b74980920210ac546c464b P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.330729 8190 raft_consensus.cc:348] T 686a74b8c1034112b293c4c7c6bcd99a P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:06.330981 8190 raft_consensus.cc:374] T 686a74b8c1034112b293c4c7c6bcd99a P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.331094 8190 raft_consensus.cc:729] T 686a74b8c1034112b293c4c7c6bcd99a P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.331696 8190 consensus_queue.cc:260] T 686a74b8c1034112b293c4c7c6bcd99a P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:06.333163 8190 ts_tablet_manager.cc:1398] T 686a74b8c1034112b293c4c7c6bcd99a P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:06.333406 8190 tablet_bootstrap.cc:492] T acdc893a70224a7b82083e998910d6ef P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:06.336795 7349 raft_consensus.cc:1261] T e0e436a62aab4f8ea5851d7c5d821fee P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:06.337347 8168 consensus_queue.cc:1035] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:06.338268 7483 raft_consensus.cc:3037] T e0e436a62aab4f8ea5851d7c5d821fee P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.338481 8205 tablet_bootstrap.cc:492] T 9a2616aa29b74980920210ac546c464b P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:06.340189 8205 tablet_bootstrap.cc:654] T 9a2616aa29b74980920210ac546c464b P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.341221 7483 raft_consensus.cc:1261] T e0e436a62aab4f8ea5851d7c5d821fee P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:06.341352 8190 tablet_bootstrap.cc:654] T acdc893a70224a7b82083e998910d6ef P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.341820 8243 consensus_queue.cc:1035] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:06.351158 8250 tablet_bootstrap.cc:492] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:06.355386 8250 tablet_bootstrap.cc:654] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.369084 8205 tablet_bootstrap.cc:492] T 9a2616aa29b74980920210ac546c464b P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:06.369416 8205 ts_tablet_manager.cc:1367] T 9a2616aa29b74980920210ac546c464b P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.003s
I20240502 21:27:06.372159 8205 raft_consensus.cc:348] T 9a2616aa29b74980920210ac546c464b P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:06.372458 8205 raft_consensus.cc:374] T 9a2616aa29b74980920210ac546c464b P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.372535 8205 raft_consensus.cc:729] T 9a2616aa29b74980920210ac546c464b P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.372793 8205 consensus_queue.cc:260] T 9a2616aa29b74980920210ac546c464b P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:06.373119 8190 tablet_bootstrap.cc:492] T acdc893a70224a7b82083e998910d6ef P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:06.373291 8190 ts_tablet_manager.cc:1367] T acdc893a70224a7b82083e998910d6ef P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.000s
I20240502 21:27:06.373625 8205 ts_tablet_manager.cc:1398] T 9a2616aa29b74980920210ac546c464b P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 21:27:06.374008 8205 tablet_bootstrap.cc:492] T 686a74b8c1034112b293c4c7c6bcd99a P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:06.374248 8190 raft_consensus.cc:348] T acdc893a70224a7b82083e998910d6ef P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:06.374522 8190 raft_consensus.cc:374] T acdc893a70224a7b82083e998910d6ef P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.376302 8260 tablet.cc:1722] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:06.376905 8260 tablet.cc:1620] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:06.374804 8190 raft_consensus.cc:729] T acdc893a70224a7b82083e998910d6ef P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.379206 8190 consensus_queue.cc:260] T acdc893a70224a7b82083e998910d6ef P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:06.380015 8190 ts_tablet_manager.cc:1398] T acdc893a70224a7b82083e998910d6ef P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240502 21:27:06.380300 8190 tablet_bootstrap.cc:492] T 9a2616aa29b74980920210ac546c464b P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:06.382671 8190 tablet_bootstrap.cc:654] T 9a2616aa29b74980920210ac546c464b P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.384716 8205 tablet_bootstrap.cc:654] T 686a74b8c1034112b293c4c7c6bcd99a P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.387995 8262 tablet.cc:1722] T e0e436a62aab4f8ea5851d7c5d821fee P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:06.388670 8262 tablet.cc:1620] T e0e436a62aab4f8ea5851d7c5d821fee P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:06.390386 8250 tablet_bootstrap.cc:492] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:06.390640 8250 ts_tablet_manager.cc:1367] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.013s
I20240502 21:27:06.391407 8261 tablet.cc:1722] T e0e436a62aab4f8ea5851d7c5d821fee P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:06.391763 8250 raft_consensus.cc:348] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:06.392063 8250 raft_consensus.cc:374] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.392222 8250 raft_consensus.cc:729] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.392532 8250 consensus_queue.cc:260] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:06.394847 8261 tablet.cc:1620] T e0e436a62aab4f8ea5851d7c5d821fee P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:06.395128 8250 ts_tablet_manager.cc:1398] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:06.395463 8250 tablet_bootstrap.cc:492] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:06.396534 8251 tablet_bootstrap.cc:492] T 9a2616aa29b74980920210ac546c464b P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:06.398375 4300 catalog_manager.cc:5576] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719) does not have the latest schema for tablet e0e436a62aab4f8ea5851d7c5d821fee (table rebalancer_dynamic_table [id=a4a396e57c1f40dc8d7cd7d2935758e9]). Expected version 1 got 0
I20240502 21:27:06.407856 8251 ts_tablet_manager.cc:1367] T 9a2616aa29b74980920210ac546c464b P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.085s user 0.011s sys 0.010s
I20240502 21:27:06.410928 8251 raft_consensus.cc:348] T 9a2616aa29b74980920210ac546c464b P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:06.411927 8205 tablet_bootstrap.cc:492] T 686a74b8c1034112b293c4c7c6bcd99a P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:06.412109 8205 ts_tablet_manager.cc:1367] T 686a74b8c1034112b293c4c7c6bcd99a P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.001s
I20240502 21:27:06.413452 8205 raft_consensus.cc:348] T 686a74b8c1034112b293c4c7c6bcd99a P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:06.413825 8205 raft_consensus.cc:374] T 686a74b8c1034112b293c4c7c6bcd99a P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.414011 8205 raft_consensus.cc:729] T 686a74b8c1034112b293c4c7c6bcd99a P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.414287 8205 consensus_queue.cc:260] T 686a74b8c1034112b293c4c7c6bcd99a P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:06.414717 8251 raft_consensus.cc:374] T 9a2616aa29b74980920210ac546c464b P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.415159 8251 raft_consensus.cc:729] T 9a2616aa29b74980920210ac546c464b P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.415230 8205 ts_tablet_manager.cc:1398] T 686a74b8c1034112b293c4c7c6bcd99a P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:27:06.415519 8205 tablet_bootstrap.cc:492] T acdc893a70224a7b82083e998910d6ef P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:06.415562 8251 consensus_queue.cc:260] T 9a2616aa29b74980920210ac546c464b P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:06.417137 8251 ts_tablet_manager.cc:1398] T 9a2616aa29b74980920210ac546c464b P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240502 21:27:06.418071 4292 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=a4a396e57c1f40dc8d7cd7d2935758e9] alter complete (version 1)
I20240502 21:27:06.421077 8190 tablet_bootstrap.cc:492] T 9a2616aa29b74980920210ac546c464b P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:06.421238 8190 ts_tablet_manager.cc:1367] T 9a2616aa29b74980920210ac546c464b P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.041s user 0.001s sys 0.005s
I20240502 21:27:06.422470 8190 raft_consensus.cc:348] T 9a2616aa29b74980920210ac546c464b P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:06.422806 8190 raft_consensus.cc:374] T 9a2616aa29b74980920210ac546c464b P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.422945 8190 raft_consensus.cc:729] T 9a2616aa29b74980920210ac546c464b P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.423179 8190 consensus_queue.cc:260] T 9a2616aa29b74980920210ac546c464b P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:06.424086 8190 ts_tablet_manager.cc:1398] T 9a2616aa29b74980920210ac546c464b P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240502 21:27:06.431110 8250 tablet_bootstrap.cc:654] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.434702 8205 tablet_bootstrap.cc:654] T acdc893a70224a7b82083e998910d6ef P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.439642 8250 tablet_bootstrap.cc:492] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:06.441385 8250 ts_tablet_manager.cc:1367] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.046s user 0.006s sys 0.000s
I20240502 21:27:06.442211 8250 raft_consensus.cc:348] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:06.442452 8250 raft_consensus.cc:374] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.442530 8250 raft_consensus.cc:729] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.442766 8250 consensus_queue.cc:260] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:06.443441 8250 ts_tablet_manager.cc:1398] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:06.446660 8205 tablet_bootstrap.cc:492] T acdc893a70224a7b82083e998910d6ef P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:06.446879 8205 ts_tablet_manager.cc:1367] T acdc893a70224a7b82083e998910d6ef P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.004s
I20240502 21:27:06.447726 8205 raft_consensus.cc:348] T acdc893a70224a7b82083e998910d6ef P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:06.447971 8205 raft_consensus.cc:374] T acdc893a70224a7b82083e998910d6ef P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.448076 8205 raft_consensus.cc:729] T acdc893a70224a7b82083e998910d6ef P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.448346 8205 consensus_queue.cc:260] T acdc893a70224a7b82083e998910d6ef P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:06.449249 8205 ts_tablet_manager.cc:1398] T acdc893a70224a7b82083e998910d6ef P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:06.481037 8120 raft_consensus.cc:482] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:06.481206 8120 raft_consensus.cc:504] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:06.481915 8120 leader_election.cc:288] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:06.482400 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "686a74b8c1034112b293c4c7c6bcd99a" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:06.482463 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "686a74b8c1034112b293c4c7c6bcd99a" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:06.482715 7483 raft_consensus.cc:2454] T 686a74b8c1034112b293c4c7c6bcd99a P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:27:06.482712 7349 raft_consensus.cc:2454] T 686a74b8c1034112b293c4c7c6bcd99a P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:27:06.483147 4770 leader_election.cc:302] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:06.483430 8120 raft_consensus.cc:2781] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:06.483536 8120 raft_consensus.cc:482] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:06.483655 8120 raft_consensus.cc:3037] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.485594 8120 raft_consensus.cc:504] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:06.486238 8120 leader_election.cc:288] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:06.486626 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "686a74b8c1034112b293c4c7c6bcd99a" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:06.486855 7483 raft_consensus.cc:3037] T 686a74b8c1034112b293c4c7c6bcd99a P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.488350 8120 raft_consensus.cc:482] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:06.488559 8120 raft_consensus.cc:504] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:06.489277 8120 leader_election.cc:288] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:06.489686 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acdc893a70224a7b82083e998910d6ef" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:06.489878 7350 raft_consensus.cc:2454] T acdc893a70224a7b82083e998910d6ef P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:27:06.490281 4772 leader_election.cc:302] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:06.490631 8120 raft_consensus.cc:2781] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:06.490649 7483 raft_consensus.cc:2454] T 686a74b8c1034112b293c4c7c6bcd99a P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:27:06.490765 8120 raft_consensus.cc:482] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:06.490850 8120 raft_consensus.cc:3037] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.492122 4770 leader_election.cc:302] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:06.492322 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acdc893a70224a7b82083e998910d6ef" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:06.486894 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "686a74b8c1034112b293c4c7c6bcd99a" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:06.492578 7483 raft_consensus.cc:2454] T acdc893a70224a7b82083e998910d6ef P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:27:06.492643 7349 raft_consensus.cc:3037] T 686a74b8c1034112b293c4c7c6bcd99a P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.493412 8024 raft_consensus.cc:2781] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:06.493824 8024 raft_consensus.cc:686] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:27:06.494122 8120 raft_consensus.cc:504] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:06.494163 8024 consensus_queue.cc:237] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:06.494920 8120 leader_election.cc:288] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:06.495246 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acdc893a70224a7b82083e998910d6ef" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:06.495405 7483 raft_consensus.cc:3037] T acdc893a70224a7b82083e998910d6ef P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.496670 7349 raft_consensus.cc:2454] T 686a74b8c1034112b293c4c7c6bcd99a P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:27:06.497521 7483 raft_consensus.cc:2454] T acdc893a70224a7b82083e998910d6ef P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:27:06.497943 4770 leader_election.cc:302] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:06.498188 8024 raft_consensus.cc:2781] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:06.498385 8024 raft_consensus.cc:686] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:27:06.498642 8024 consensus_queue.cc:237] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:06.500209 4294 catalog_manager.cc:5484] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:06.508183 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acdc893a70224a7b82083e998910d6ef" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:06.508399 7349 raft_consensus.cc:3037] T acdc893a70224a7b82083e998910d6ef P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.509258 4294 catalog_manager.cc:5484] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:06.511119 7349 raft_consensus.cc:2454] T acdc893a70224a7b82083e998910d6ef P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:27:06.525102 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:06.532420 8260 tablet.cc:1722] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:06.532893 8260 tablet.cc:1620] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:06.533025 8262 tablet.cc:1722] T e0e436a62aab4f8ea5851d7c5d821fee P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:06.533491 8262 tablet.cc:1620] T e0e436a62aab4f8ea5851d7c5d821fee P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:06.534019 8261 tablet.cc:1722] T e0e436a62aab4f8ea5851d7c5d821fee P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:06.534543 8261 tablet.cc:1620] T e0e436a62aab4f8ea5851d7c5d821fee P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:06.542501 4300 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=a4a396e57c1f40dc8d7cd7d2935758e9] alter complete (version 2)
I20240502 21:27:06.569537 8074 raft_consensus.cc:482] T 9a2616aa29b74980920210ac546c464b P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:06.569729 8074 raft_consensus.cc:504] T 9a2616aa29b74980920210ac546c464b P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:06.570375 8074 leader_election.cc:288] T 9a2616aa29b74980920210ac546c464b P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:06.570757 7617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a2616aa29b74980920210ac546c464b" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:06.570979 7617 raft_consensus.cc:2454] T 9a2616aa29b74980920210ac546c464b P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 0.
I20240502 21:27:06.570932 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a2616aa29b74980920210ac546c464b" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:06.571197 7483 raft_consensus.cc:2454] T 9a2616aa29b74980920210ac546c464b P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 0.
I20240502 21:27:06.571445 7282 leader_election.cc:302] T 9a2616aa29b74980920210ac546c464b P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:06.571923 8074 raft_consensus.cc:2781] T 9a2616aa29b74980920210ac546c464b P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:06.572038 8074 raft_consensus.cc:482] T 9a2616aa29b74980920210ac546c464b P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:06.572156 8074 raft_consensus.cc:3037] T 9a2616aa29b74980920210ac546c464b P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.574246 8074 raft_consensus.cc:504] T 9a2616aa29b74980920210ac546c464b P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:06.574831 8074 leader_election.cc:288] T 9a2616aa29b74980920210ac546c464b P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Requested vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:06.575297 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a2616aa29b74980920210ac546c464b" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:06.575299 7617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a2616aa29b74980920210ac546c464b" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:06.575479 7483 raft_consensus.cc:3037] T 9a2616aa29b74980920210ac546c464b P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.575485 7617 raft_consensus.cc:3037] T 9a2616aa29b74980920210ac546c464b P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.577688 7617 raft_consensus.cc:2454] T 9a2616aa29b74980920210ac546c464b P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
I20240502 21:27:06.577772 7483 raft_consensus.cc:2454] T 9a2616aa29b74980920210ac546c464b P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
I20240502 21:27:06.578261 7282 leader_election.cc:302] T 9a2616aa29b74980920210ac546c464b P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:06.578593 8074 raft_consensus.cc:2781] T 9a2616aa29b74980920210ac546c464b P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:06.578981 8074 raft_consensus.cc:686] T 9a2616aa29b74980920210ac546c464b P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Becoming Leader. State: Replica: d0456c5e55a542128688904e003d403c, State: Running, Role: LEADER
I20240502 21:27:06.579284 8074 consensus_queue.cc:237] T 9a2616aa29b74980920210ac546c464b P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:06.582463 4294 catalog_manager.cc:5484] T 9a2616aa29b74980920210ac546c464b P d0456c5e55a542128688904e003d403c reported cstate change: term changed from 0 to 1, leader changed from <none> to d0456c5e55a542128688904e003d403c (127.29.82.70). New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:06.645934 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:06.653153 8260 tablet.cc:1722] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:06.654029 8261 tablet.cc:1722] T e0e436a62aab4f8ea5851d7c5d821fee P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:06.654086 8262 tablet.cc:1722] T e0e436a62aab4f8ea5851d7c5d821fee P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:06.663508 4294 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=a4a396e57c1f40dc8d7cd7d2935758e9] alter complete (version 3)
I20240502 21:27:06.769001 4300 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:06.769264 4300 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:06.772369 4300 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet e0e436a62aab4f8ea5851d7c5d821fee
I20240502 21:27:06.773257 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet e0e436a62aab4f8ea5851d7c5d821fee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:06 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:06.773334 7462 tablet_service.cc:1514] Processing DeleteTablet for tablet e0e436a62aab4f8ea5851d7c5d821fee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:06 UTC) from {username='slave'} at 127.0.0.1:37182
I20240502 21:27:06.773581 7330 tablet_service.cc:1514] Processing DeleteTablet for tablet e0e436a62aab4f8ea5851d7c5d821fee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:06 UTC) from {username='slave'} at 127.0.0.1:47078
I20240502 21:27:06.786108 8280 tablet_replica.cc:330] T e0e436a62aab4f8ea5851d7c5d821fee P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:27:06.786188 8278 tablet_replica.cc:330] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:06.786474 8278 raft_consensus.cc:2229] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:06.786760 8280 raft_consensus.cc:2229] T e0e436a62aab4f8ea5851d7c5d821fee P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:06.786978 8278 raft_consensus.cc:2258] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:06.787165 8280 raft_consensus.cc:2258] T e0e436a62aab4f8ea5851d7c5d821fee P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:06.787998 8278 ts_tablet_manager.cc:1875] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:06.788394 8280 ts_tablet_manager.cc:1875] T e0e436a62aab4f8ea5851d7c5d821fee P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:06.788542 8279 tablet_replica.cc:330] T e0e436a62aab4f8ea5851d7c5d821fee P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:27:06.788775 8279 raft_consensus.cc:2229] T e0e436a62aab4f8ea5851d7c5d821fee P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:06.789005 8279 raft_consensus.cc:2258] T e0e436a62aab4f8ea5851d7c5d821fee P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:06.790007 8279 ts_tablet_manager.cc:1875] T e0e436a62aab4f8ea5851d7c5d821fee P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:06.793934 8280 ts_tablet_manager.cc:1888] T e0e436a62aab4f8ea5851d7c5d821fee P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:06.794054 8280 log.cc:1196] T e0e436a62aab4f8ea5851d7c5d821fee P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/e0e436a62aab4f8ea5851d7c5d821fee
I20240502 21:27:06.794565 8280 ts_tablet_manager.cc:1909] T e0e436a62aab4f8ea5851d7c5d821fee P d0456c5e55a542128688904e003d403c: Deleting consensus metadata
I20240502 21:27:06.795182 8278 ts_tablet_manager.cc:1888] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:06.795306 8278 log.cc:1196] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/e0e436a62aab4f8ea5851d7c5d821fee
I20240502 21:27:06.795732 8278 ts_tablet_manager.cc:1909] T e0e436a62aab4f8ea5851d7c5d821fee P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:06.796675 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet e0e436a62aab4f8ea5851d7c5d821fee (table rebalancer_dynamic_table_new_name [id=a4a396e57c1f40dc8d7cd7d2935758e9]) successfully deleted
I20240502 21:27:06.796933 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet e0e436a62aab4f8ea5851d7c5d821fee (table rebalancer_dynamic_table_new_name [id=a4a396e57c1f40dc8d7cd7d2935758e9]) successfully deleted
I20240502 21:27:06.797464 8279 ts_tablet_manager.cc:1888] T e0e436a62aab4f8ea5851d7c5d821fee P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:06.797581 8279 log.cc:1196] T e0e436a62aab4f8ea5851d7c5d821fee P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/e0e436a62aab4f8ea5851d7c5d821fee
I20240502 21:27:06.797977 8279 ts_tablet_manager.cc:1909] T e0e436a62aab4f8ea5851d7c5d821fee P 31a72286397c4edc9a63eca3412be168: Deleting consensus metadata
I20240502 21:27:06.799153 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet e0e436a62aab4f8ea5851d7c5d821fee (table rebalancer_dynamic_table_new_name [id=a4a396e57c1f40dc8d7cd7d2935758e9]) successfully deleted
I20240502 21:27:06.876653 4300 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:06.884052 7596 tablet_service.cc:1467] Processing CreateTablet for tablet 433812854a7044098e0ec099134db404 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=de0655191f7241449a633da1f8ff1c13]), partition=
I20240502 21:27:06.884385 7330 tablet_service.cc:1467] Processing CreateTablet for tablet 433812854a7044098e0ec099134db404 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=de0655191f7241449a633da1f8ff1c13]), partition=
I20240502 21:27:06.884429 7596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 433812854a7044098e0ec099134db404. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.884737 7330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 433812854a7044098e0ec099134db404. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.889387 8205 tablet_bootstrap.cc:492] T 433812854a7044098e0ec099134db404 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:06.889990 7462 tablet_service.cc:1467] Processing CreateTablet for tablet 433812854a7044098e0ec099134db404 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=de0655191f7241449a633da1f8ff1c13]), partition=
I20240502 21:27:06.890437 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 433812854a7044098e0ec099134db404. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.890956 8205 tablet_bootstrap.cc:654] T 433812854a7044098e0ec099134db404 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.892843 8205 tablet_bootstrap.cc:492] T 433812854a7044098e0ec099134db404 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:06.892985 8205 ts_tablet_manager.cc:1367] T 433812854a7044098e0ec099134db404 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:27:06.893810 8205 raft_consensus.cc:348] T 433812854a7044098e0ec099134db404 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:06.894037 8205 raft_consensus.cc:374] T 433812854a7044098e0ec099134db404 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.894131 8205 raft_consensus.cc:729] T 433812854a7044098e0ec099134db404 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.894371 8205 consensus_queue.cc:260] T 433812854a7044098e0ec099134db404 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:06.895090 8205 ts_tablet_manager.cc:1398] T 433812854a7044098e0ec099134db404 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:06.897845 8251 tablet_bootstrap.cc:492] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:06.899142 8251 tablet_bootstrap.cc:654] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.901324 8251 tablet_bootstrap.cc:492] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:06.901497 8251 ts_tablet_manager.cc:1367] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:27:06.902081 8251 raft_consensus.cc:348] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:06.902237 8251 raft_consensus.cc:374] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.902284 8251 raft_consensus.cc:729] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.902472 8251 consensus_queue.cc:260] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:06.902933 8251 ts_tablet_manager.cc:1398] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:27:06.904034 8190 tablet_bootstrap.cc:492] T 433812854a7044098e0ec099134db404 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:06.905489 8190 tablet_bootstrap.cc:654] T 433812854a7044098e0ec099134db404 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.907289 8190 tablet_bootstrap.cc:492] T 433812854a7044098e0ec099134db404 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:06.907459 8190 ts_tablet_manager.cc:1367] T 433812854a7044098e0ec099134db404 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:27:06.908079 8190 raft_consensus.cc:348] T 433812854a7044098e0ec099134db404 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:06.908258 8190 raft_consensus.cc:374] T 433812854a7044098e0ec099134db404 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.908315 8190 raft_consensus.cc:729] T 433812854a7044098e0ec099134db404 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.908541 8190 consensus_queue.cc:260] T 433812854a7044098e0ec099134db404 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:06.909013 8190 ts_tablet_manager.cc:1398] T 433812854a7044098e0ec099134db404 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 21:27:06.939265 8270 consensus_queue.cc:1035] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:06.945953 8270 consensus_queue.cc:1035] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:07.024367 8275 raft_consensus.cc:482] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:07.024591 8275 raft_consensus.cc:504] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:07.025288 8275 leader_election.cc:288] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:07.025746 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433812854a7044098e0ec099134db404" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:07.025776 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433812854a7044098e0ec099134db404" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:07.025946 7483 raft_consensus.cc:2454] T 433812854a7044098e0ec099134db404 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:07.025997 7349 raft_consensus.cc:2454] T 433812854a7044098e0ec099134db404 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:07.026393 7549 leader_election.cc:302] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168; no voters:
I20240502 21:27:07.026671 8275 raft_consensus.cc:2781] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:07.026791 8275 raft_consensus.cc:482] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:07.026902 8275 raft_consensus.cc:3037] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:07.030050 8275 raft_consensus.cc:504] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:07.030797 8275 leader_election.cc:288] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Requested vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:07.031121 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433812854a7044098e0ec099134db404" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:07.031176 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433812854a7044098e0ec099134db404" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:07.031304 7483 raft_consensus.cc:3037] T 433812854a7044098e0ec099134db404 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:07.031386 7349 raft_consensus.cc:3037] T 433812854a7044098e0ec099134db404 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:07.033476 7483 raft_consensus.cc:2454] T 433812854a7044098e0ec099134db404 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:07.033603 7349 raft_consensus.cc:2454] T 433812854a7044098e0ec099134db404 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:07.033856 7549 leader_election.cc:302] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168; no voters:
I20240502 21:27:07.034224 8275 raft_consensus.cc:2781] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:07.034396 8275 raft_consensus.cc:686] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Becoming Leader. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Running, Role: LEADER
I20240502 21:27:07.035172 8275 consensus_queue.cc:237] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:07.038565 4300 catalog_manager.cc:5484] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ddd486109f5434cace13d60d633ac04 (127.29.82.72). New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:07.047250 8270 consensus_queue.cc:1035] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:07.052814 8270 consensus_queue.cc:1035] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:07.074913 8074 consensus_queue.cc:1035] T 9a2616aa29b74980920210ac546c464b P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:07.078843 8276 consensus_queue.cc:1035] T 9a2616aa29b74980920210ac546c464b P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:07.108408 4300 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:07.117256 4686 tablet_service.cc:1467] Processing CreateTablet for tablet d09a80f79473469e92310ded0f4edfdd (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=1488a4e68a204d09a88b4a05c1c2cc24]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:07.117794 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d09a80f79473469e92310ded0f4edfdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.117681 7462 tablet_service.cc:1467] Processing CreateTablet for tablet d09a80f79473469e92310ded0f4edfdd (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=1488a4e68a204d09a88b4a05c1c2cc24]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:07.117758 7464 tablet_service.cc:1467] Processing CreateTablet for tablet 6065bdba6e2840b8b368d8f92d1fa260 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=1488a4e68a204d09a88b4a05c1c2cc24]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:07.118155 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d09a80f79473469e92310ded0f4edfdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.118446 7464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6065bdba6e2840b8b368d8f92d1fa260. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.120297 7595 tablet_service.cc:1467] Processing CreateTablet for tablet 6065bdba6e2840b8b368d8f92d1fa260 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=1488a4e68a204d09a88b4a05c1c2cc24]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:07.120361 7194 tablet_service.cc:1467] Processing CreateTablet for tablet 6065bdba6e2840b8b368d8f92d1fa260 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=1488a4e68a204d09a88b4a05c1c2cc24]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:07.120754 7595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6065bdba6e2840b8b368d8f92d1fa260. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.120827 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6065bdba6e2840b8b368d8f92d1fa260. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.123971 7596 tablet_service.cc:1467] Processing CreateTablet for tablet d09a80f79473469e92310ded0f4edfdd (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=1488a4e68a204d09a88b4a05c1c2cc24]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:07.124363 7596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d09a80f79473469e92310ded0f4edfdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.126768 7597 tablet_service.cc:1467] Processing CreateTablet for tablet 063bdcf9cea74d5cbe3143eb16fba6a5 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=1488a4e68a204d09a88b4a05c1c2cc24]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:07.127144 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 063bdcf9cea74d5cbe3143eb16fba6a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.130690 7463 tablet_service.cc:1467] Processing CreateTablet for tablet 063bdcf9cea74d5cbe3143eb16fba6a5 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=1488a4e68a204d09a88b4a05c1c2cc24]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:07.131110 7463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 063bdcf9cea74d5cbe3143eb16fba6a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.133843 7195 tablet_service.cc:1467] Processing CreateTablet for tablet 063bdcf9cea74d5cbe3143eb16fba6a5 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=1488a4e68a204d09a88b4a05c1c2cc24]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:07.134299 7195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 063bdcf9cea74d5cbe3143eb16fba6a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.136607 8190 tablet_bootstrap.cc:492] T d09a80f79473469e92310ded0f4edfdd P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:07.138960 8190 tablet_bootstrap.cc:654] T d09a80f79473469e92310ded0f4edfdd P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:07.142187 8190 tablet_bootstrap.cc:492] T d09a80f79473469e92310ded0f4edfdd P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:07.142387 8190 ts_tablet_manager.cc:1367] T d09a80f79473469e92310ded0f4edfdd P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 21:27:07.143373 8190 raft_consensus.cc:348] T d09a80f79473469e92310ded0f4edfdd P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:07.143685 8190 raft_consensus.cc:374] T d09a80f79473469e92310ded0f4edfdd P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:07.143812 8190 raft_consensus.cc:729] T d09a80f79473469e92310ded0f4edfdd P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:07.144095 8190 consensus_queue.cc:260] T d09a80f79473469e92310ded0f4edfdd P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:07.146159 8190 ts_tablet_manager.cc:1398] T d09a80f79473469e92310ded0f4edfdd P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:27:07.146507 8190 tablet_bootstrap.cc:492] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:07.149456 8251 tablet_bootstrap.cc:492] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:07.150040 8190 tablet_bootstrap.cc:654] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:07.151151 8251 tablet_bootstrap.cc:654] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:07.153288 4291 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:07.154153 8296 tablet_bootstrap.cc:492] T 6065bdba6e2840b8b368d8f92d1fa260 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:07.155167 8295 tablet_bootstrap.cc:492] T d09a80f79473469e92310ded0f4edfdd P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:07.155259 8251 tablet_bootstrap.cc:492] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:07.155400 8251 ts_tablet_manager.cc:1367] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 21:27:07.156229 8251 raft_consensus.cc:348] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:07.156421 8251 raft_consensus.cc:374] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:07.156538 8251 raft_consensus.cc:729] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:07.156769 8251 consensus_queue.cc:260] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:07.157346 8251 ts_tablet_manager.cc:1398] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:07.157624 8251 tablet_bootstrap.cc:492] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:07.157642 8295 tablet_bootstrap.cc:654] T d09a80f79473469e92310ded0f4edfdd P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:07.160840 8251 tablet_bootstrap.cc:654] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:07.164237 7349 raft_consensus.cc:1261] T 433812854a7044098e0ec099134db404 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:07.164881 8275 consensus_queue.cc:1035] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:07.165426 7483 raft_consensus.cc:1261] T 433812854a7044098e0ec099134db404 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:07.165977 8190 tablet_bootstrap.cc:492] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:07.166116 8190 ts_tablet_manager.cc:1367] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.003s
I20240502 21:27:07.167297 8275 consensus_queue.cc:1035] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:07.166997 8190 raft_consensus.cc:348] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:07.167515 8190 raft_consensus.cc:374] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:07.167609 8190 raft_consensus.cc:729] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:07.167883 8190 consensus_queue.cc:260] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:07.168140 8296 tablet_bootstrap.cc:654] T 6065bdba6e2840b8b368d8f92d1fa260 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:07.168540 8190 ts_tablet_manager.cc:1398] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:07.168871 8190 tablet_bootstrap.cc:492] T 6065bdba6e2840b8b368d8f92d1fa260 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:07.170740 8190 tablet_bootstrap.cc:654] T 6065bdba6e2840b8b368d8f92d1fa260 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:07.173153 8251 tablet_bootstrap.cc:492] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:07.173318 8251 ts_tablet_manager.cc:1367] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.002s
I20240502 21:27:07.175087 8251 raft_consensus.cc:348] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:07.175331 8251 raft_consensus.cc:374] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:07.175415 8251 raft_consensus.cc:729] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:07.175732 8251 consensus_queue.cc:260] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:07.176874 8295 tablet_bootstrap.cc:492] T d09a80f79473469e92310ded0f4edfdd P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:27:07.177510 8295 ts_tablet_manager.cc:1367] T d09a80f79473469e92310ded0f4edfdd P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.008s
I20240502 21:27:07.181277 8251 ts_tablet_manager.cc:1398] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 21:27:07.183002 8251 tablet_bootstrap.cc:492] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:07.179677 8295 raft_consensus.cc:348] T d09a80f79473469e92310ded0f4edfdd P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:07.183949 8295 raft_consensus.cc:374] T d09a80f79473469e92310ded0f4edfdd P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:07.184309 8295 raft_consensus.cc:729] T d09a80f79473469e92310ded0f4edfdd P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:07.184785 8251 tablet_bootstrap.cc:654] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:07.185045 8295 consensus_queue.cc:260] T d09a80f79473469e92310ded0f4edfdd P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:07.191417 8295 ts_tablet_manager.cc:1398] T d09a80f79473469e92310ded0f4edfdd P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240502 21:27:07.194209 8190 tablet_bootstrap.cc:492] T 6065bdba6e2840b8b368d8f92d1fa260 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:07.194482 8190 ts_tablet_manager.cc:1367] T 6065bdba6e2840b8b368d8f92d1fa260 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.004s
I20240502 21:27:07.196946 8296 tablet_bootstrap.cc:492] T 6065bdba6e2840b8b368d8f92d1fa260 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:07.196925 8190 raft_consensus.cc:348] T 6065bdba6e2840b8b368d8f92d1fa260 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:07.197233 8296 ts_tablet_manager.cc:1367] T 6065bdba6e2840b8b368d8f92d1fa260 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.043s user 0.005s sys 0.020s
I20240502 21:27:07.197410 8190 raft_consensus.cc:374] T 6065bdba6e2840b8b368d8f92d1fa260 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:07.197587 8190 raft_consensus.cc:729] T 6065bdba6e2840b8b368d8f92d1fa260 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:07.197901 8190 consensus_queue.cc:260] T 6065bdba6e2840b8b368d8f92d1fa260 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:07.198092 8296 raft_consensus.cc:348] T 6065bdba6e2840b8b368d8f92d1fa260 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:07.198318 8296 raft_consensus.cc:374] T 6065bdba6e2840b8b368d8f92d1fa260 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:07.198424 8296 raft_consensus.cc:729] T 6065bdba6e2840b8b368d8f92d1fa260 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:07.198696 8296 consensus_queue.cc:260] T 6065bdba6e2840b8b368d8f92d1fa260 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:07.198828 8190 ts_tablet_manager.cc:1398] T 6065bdba6e2840b8b368d8f92d1fa260 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 21:27:07.200897 8296 ts_tablet_manager.cc:1398] T 6065bdba6e2840b8b368d8f92d1fa260 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 21:27:07.201216 8296 tablet_bootstrap.cc:492] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:07.202828 8296 tablet_bootstrap.cc:654] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:07.203717 8251 tablet_bootstrap.cc:492] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:07.203891 8251 ts_tablet_manager.cc:1367] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.001s
I20240502 21:27:07.204793 8251 raft_consensus.cc:348] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:07.205063 8251 raft_consensus.cc:374] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:07.205188 8251 raft_consensus.cc:729] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:07.205490 8251 consensus_queue.cc:260] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:07.207978 8251 ts_tablet_manager.cc:1398] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:07.213187 8296 tablet_bootstrap.cc:492] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:07.213348 8296 ts_tablet_manager.cc:1367] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20240502 21:27:07.214051 8296 raft_consensus.cc:348] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:07.214246 8296 raft_consensus.cc:374] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:07.214330 8296 raft_consensus.cc:729] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:07.214577 8296 consensus_queue.cc:260] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:07.215212 8296 ts_tablet_manager.cc:1398] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:07.217658 8306 tablet.cc:1722] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:07.218326 8306 tablet.cc:1620] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04: MemRowSet was empty: no flush needed.
I20240502 21:27:07.220098 8307 tablet.cc:1722] T 433812854a7044098e0ec099134db404 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:07.220703 8307 tablet.cc:1620] T 433812854a7044098e0ec099134db404 P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:07.225688 4300 catalog_manager.cc:5576] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913) does not have the latest schema for tablet 433812854a7044098e0ec099134db404 (table rebalancer_dynamic_table [id=de0655191f7241449a633da1f8ff1c13]). Expected version 1 got 0
I20240502 21:27:07.230732 4291 catalog_manager.cc:5576] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719) does not have the latest schema for tablet 433812854a7044098e0ec099134db404 (table rebalancer_dynamic_table [id=de0655191f7241449a633da1f8ff1c13]). Expected version 1 got 0
I20240502 21:27:07.233239 4294 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=de0655191f7241449a633da1f8ff1c13] alter complete (version 1)
I20240502 21:27:07.240008 8311 tablet.cc:1722] T 433812854a7044098e0ec099134db404 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:07.240759 8311 tablet.cc:1620] T 433812854a7044098e0ec099134db404 P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:07.242079 4291 catalog_manager.cc:5576] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913) does not have the latest schema for tablet 433812854a7044098e0ec099134db404 (table rebalancer_dynamic_table [id=de0655191f7241449a633da1f8ff1c13]). Expected version 1 got 0
I20240502 21:27:07.251591 8275 raft_consensus.cc:482] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:07.251809 8275 raft_consensus.cc:504] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:07.253650 8275 leader_election.cc:288] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:07.254242 8303 raft_consensus.cc:482] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:07.254264 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6065bdba6e2840b8b368d8f92d1fa260" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:07.254285 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6065bdba6e2840b8b368d8f92d1fa260" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:07.254390 8303 raft_consensus.cc:504] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:07.254479 7215 raft_consensus.cc:2454] T 6065bdba6e2840b8b368d8f92d1fa260 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:07.254555 7484 raft_consensus.cc:2454] T 6065bdba6e2840b8b368d8f92d1fa260 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:07.255013 8303 leader_election.cc:288] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:07.254992 7551 leader_election.cc:302] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:07.255380 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "063bdcf9cea74d5cbe3143eb16fba6a5" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:07.255632 7618 raft_consensus.cc:2454] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:07.255710 8275 raft_consensus.cc:2781] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:07.255789 8275 raft_consensus.cc:482] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:07.255836 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "063bdcf9cea74d5cbe3143eb16fba6a5" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:07.255929 8275 raft_consensus.cc:3037] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:07.256038 7215 raft_consensus.cc:2454] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:07.256743 7418 leader_election.cc:302] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168; no voters:
I20240502 21:27:07.257174 8303 raft_consensus.cc:2781] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:07.257280 8303 raft_consensus.cc:482] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:07.257355 8303 raft_consensus.cc:3037] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:07.258616 8275 raft_consensus.cc:504] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:07.259552 8275 leader_election.cc:288] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Requested vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:07.259963 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6065bdba6e2840b8b368d8f92d1fa260" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:07.260182 7215 raft_consensus.cc:3037] T 6065bdba6e2840b8b368d8f92d1fa260 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:07.260463 8303 raft_consensus.cc:504] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:07.260965 8303 leader_election.cc:288] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Requested vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:07.262118 7215 raft_consensus.cc:2454] T 6065bdba6e2840b8b368d8f92d1fa260 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:07.262490 7551 leader_election.cc:302] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:07.262573 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "063bdcf9cea74d5cbe3143eb16fba6a5" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:07.262588 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6065bdba6e2840b8b368d8f92d1fa260" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:07.262804 7618 raft_consensus.cc:3037] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:07.262831 8275 raft_consensus.cc:2781] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:07.262897 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "063bdcf9cea74d5cbe3143eb16fba6a5" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:07.263078 7215 raft_consensus.cc:3037] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:07.263080 8275 raft_consensus.cc:686] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Becoming Leader. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Running, Role: LEADER
I20240502 21:27:07.263383 8275 consensus_queue.cc:237] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:07.266184 7215 raft_consensus.cc:2454] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:07.266625 7419 leader_election.cc:302] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:07.266664 7484 raft_consensus.cc:3037] T 6065bdba6e2840b8b368d8f92d1fa260 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:07.266990 8303 raft_consensus.cc:2781] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:07.267289 8303 raft_consensus.cc:686] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Becoming Leader. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Running, Role: LEADER
I20240502 21:27:07.267872 8303 consensus_queue.cc:237] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:07.268424 7618 raft_consensus.cc:2454] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:07.274484 7484 raft_consensus.cc:2454] T 6065bdba6e2840b8b368d8f92d1fa260 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:07.278621 4291 catalog_manager.cc:5484] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ddd486109f5434cace13d60d633ac04 (127.29.82.72). New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:07.281870 4300 catalog_manager.cc:5484] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a72286397c4edc9a63eca3412be168 (127.29.82.71). New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:07.335857 4300 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:07.341656 7942 raft_consensus.cc:482] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:07.341845 7942 raft_consensus.cc:504] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:07.342674 7942 leader_election.cc:288] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:07.346525 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09a80f79473469e92310ded0f4edfdd" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:07.346763 7483 raft_consensus.cc:2454] T d09a80f79473469e92310ded0f4edfdd P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:07.348651 7549 leader_election.cc:302] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168; no voters:
I20240502 21:27:07.349246 8307 tablet.cc:1722] T 433812854a7044098e0ec099134db404 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:07.349398 8287 raft_consensus.cc:2781] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:07.349531 8287 raft_consensus.cc:482] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:07.349673 8287 raft_consensus.cc:3037] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:07.349910 8307 tablet.cc:1620] T 433812854a7044098e0ec099134db404 P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:07.353760 8287 raft_consensus.cc:504] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:07.354952 8287 leader_election.cc:288] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:07.356454 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09a80f79473469e92310ded0f4edfdd" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:07.356792 7484 raft_consensus.cc:3037] T d09a80f79473469e92310ded0f4edfdd P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:07.360414 8311 tablet.cc:1722] T 433812854a7044098e0ec099134db404 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:07.361032 8311 tablet.cc:1620] T 433812854a7044098e0ec099134db404 P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:07.363638 4300 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=de0655191f7241449a633da1f8ff1c13] alter complete (version 2)
I20240502 21:27:07.371912 8306 tablet.cc:1722] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:07.372557 8306 tablet.cc:1620] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04: MemRowSet was empty: no flush needed.
I20240502 21:27:07.377697 7484 raft_consensus.cc:2454] T d09a80f79473469e92310ded0f4edfdd P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:07.378476 7549 leader_election.cc:302] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168; no voters:
I20240502 21:27:07.378871 7942 raft_consensus.cc:2781] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:07.379055 7942 raft_consensus.cc:686] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Becoming Leader. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Running, Role: LEADER
I20240502 21:27:07.379367 7942 consensus_queue.cc:237] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:07.384721 4291 catalog_manager.cc:5484] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ddd486109f5434cace13d60d633ac04 (127.29.82.72). New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:27:07.391366 4704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09a80f79473469e92310ded0f4edfdd" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:27:07.391409 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09a80f79473469e92310ded0f4edfdd" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:27:07.391676 4704 raft_consensus.cc:2454] T d09a80f79473469e92310ded0f4edfdd P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:07.472707 4291 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:07.480650 8306 tablet.cc:1722] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:07.481292 8311 tablet.cc:1722] T 433812854a7044098e0ec099134db404 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:07.482061 8307 tablet.cc:1722] T 433812854a7044098e0ec099134db404 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:07.490077 4300 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=de0655191f7241449a633da1f8ff1c13] alter complete (version 3)
I20240502 21:27:07.595336 4300 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:07.595558 4300 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:07.598498 4300 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 433812854a7044098e0ec099134db404
I20240502 21:27:07.599354 7595 tablet_service.cc:1514] Processing DeleteTablet for tablet 433812854a7044098e0ec099134db404 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:07 UTC) from {username='slave'} at 127.0.0.1:44444
I20240502 21:27:07.599503 7462 tablet_service.cc:1514] Processing DeleteTablet for tablet 433812854a7044098e0ec099134db404 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:07 UTC) from {username='slave'} at 127.0.0.1:37182
I20240502 21:27:07.600199 8319 tablet_replica.cc:330] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04: stopping tablet replica
I20240502 21:27:07.600203 7330 tablet_service.cc:1514] Processing DeleteTablet for tablet 433812854a7044098e0ec099134db404 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:07 UTC) from {username='slave'} at 127.0.0.1:47078
I20240502 21:27:07.600322 8320 tablet_replica.cc:330] T 433812854a7044098e0ec099134db404 P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:27:07.600466 8319 raft_consensus.cc:2229] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:07.600564 8320 raft_consensus.cc:2229] T 433812854a7044098e0ec099134db404 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:07.600785 8321 tablet_replica.cc:330] T 433812854a7044098e0ec099134db404 P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:27:07.600804 8320 raft_consensus.cc:2258] T 433812854a7044098e0ec099134db404 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:07.601050 8319 raft_consensus.cc:2258] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:07.601083 8321 raft_consensus.cc:2229] T 433812854a7044098e0ec099134db404 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:07.601352 8321 raft_consensus.cc:2258] T 433812854a7044098e0ec099134db404 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:07.601743 8320 ts_tablet_manager.cc:1875] T 433812854a7044098e0ec099134db404 P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:07.602221 8321 ts_tablet_manager.cc:1875] T 433812854a7044098e0ec099134db404 P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:07.602257 8319 ts_tablet_manager.cc:1875] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:07.606645 8319 ts_tablet_manager.cc:1888] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:07.606812 8319 log.cc:1196] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/wals/433812854a7044098e0ec099134db404
I20240502 21:27:07.607332 8319 ts_tablet_manager.cc:1909] T 433812854a7044098e0ec099134db404 P 0ddd486109f5434cace13d60d633ac04: Deleting consensus metadata
I20240502 21:27:07.608095 8320 ts_tablet_manager.cc:1888] T 433812854a7044098e0ec099134db404 P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:07.608247 8320 log.cc:1196] T 433812854a7044098e0ec099134db404 P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/433812854a7044098e0ec099134db404
I20240502 21:27:07.608517 8321 ts_tablet_manager.cc:1888] T 433812854a7044098e0ec099134db404 P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:07.608678 8321 log.cc:1196] T 433812854a7044098e0ec099134db404 P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/433812854a7044098e0ec099134db404
I20240502 21:27:07.608752 8320 ts_tablet_manager.cc:1909] T 433812854a7044098e0ec099134db404 P 31a72286397c4edc9a63eca3412be168: Deleting consensus metadata
I20240502 21:27:07.608774 4285 catalog_manager.cc:4830] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): tablet 433812854a7044098e0ec099134db404 (table rebalancer_dynamic_table_new_name [id=de0655191f7241449a633da1f8ff1c13]) successfully deleted
I20240502 21:27:07.609169 8321 ts_tablet_manager.cc:1909] T 433812854a7044098e0ec099134db404 P d0456c5e55a542128688904e003d403c: Deleting consensus metadata
I20240502 21:27:07.609877 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet 433812854a7044098e0ec099134db404 (table rebalancer_dynamic_table_new_name [id=de0655191f7241449a633da1f8ff1c13]) successfully deleted
I20240502 21:27:07.610145 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet 433812854a7044098e0ec099134db404 (table rebalancer_dynamic_table_new_name [id=de0655191f7241449a633da1f8ff1c13]) successfully deleted
I20240502 21:27:07.647847 7942 consensus_queue.cc:1035] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:07.652169 7942 consensus_queue.cc:1035] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:07.701170 4300 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:07.708434 7462 tablet_service.cc:1467] Processing CreateTablet for tablet 371146bfe9a940c98bd13143506a7322 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a9f1ef6bc55440d7a04bba730e02577e]), partition=
I20240502 21:27:07.708727 4421 tablet_service.cc:1467] Processing CreateTablet for tablet 371146bfe9a940c98bd13143506a7322 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a9f1ef6bc55440d7a04bba730e02577e]), partition=
I20240502 21:27:07.708806 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 371146bfe9a940c98bd13143506a7322. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.709107 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 371146bfe9a940c98bd13143506a7322. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.712018 4818 tablet_service.cc:1467] Processing CreateTablet for tablet 371146bfe9a940c98bd13143506a7322 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a9f1ef6bc55440d7a04bba730e02577e]), partition=
I20240502 21:27:07.712455 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 371146bfe9a940c98bd13143506a7322. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.714416 8326 tablet_bootstrap.cc:492] T 371146bfe9a940c98bd13143506a7322 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:07.716095 8326 tablet_bootstrap.cc:654] T 371146bfe9a940c98bd13143506a7322 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:07.718104 8326 tablet_bootstrap.cc:492] T 371146bfe9a940c98bd13143506a7322 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:07.718252 8326 ts_tablet_manager.cc:1367] T 371146bfe9a940c98bd13143506a7322 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:27:07.718971 8326 raft_consensus.cc:348] T 371146bfe9a940c98bd13143506a7322 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:07.719199 8326 raft_consensus.cc:374] T 371146bfe9a940c98bd13143506a7322 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:07.719283 8326 raft_consensus.cc:729] T 371146bfe9a940c98bd13143506a7322 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:07.719729 8326 consensus_queue.cc:260] T 371146bfe9a940c98bd13143506a7322 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:07.720340 8328 tablet_bootstrap.cc:492] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:07.721184 8326 ts_tablet_manager.cc:1398] T 371146bfe9a940c98bd13143506a7322 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:07.721954 8328 tablet_bootstrap.cc:654] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:07.726346 8328 tablet_bootstrap.cc:492] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:07.726492 8328 ts_tablet_manager.cc:1367] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 21:27:07.727267 8328 raft_consensus.cc:348] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:07.727535 8328 raft_consensus.cc:374] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:07.727669 8328 raft_consensus.cc:729] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:07.727967 8329 tablet_bootstrap.cc:492] T 371146bfe9a940c98bd13143506a7322 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:07.727932 8328 consensus_queue.cc:260] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:07.729463 8328 ts_tablet_manager.cc:1398] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:07.729668 8329 tablet_bootstrap.cc:654] T 371146bfe9a940c98bd13143506a7322 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:07.731591 8329 tablet_bootstrap.cc:492] T 371146bfe9a940c98bd13143506a7322 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:07.731737 8329 ts_tablet_manager.cc:1367] T 371146bfe9a940c98bd13143506a7322 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20240502 21:27:07.732383 8329 raft_consensus.cc:348] T 371146bfe9a940c98bd13143506a7322 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:07.732570 8329 raft_consensus.cc:374] T 371146bfe9a940c98bd13143506a7322 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:07.732669 8329 raft_consensus.cc:729] T 371146bfe9a940c98bd13143506a7322 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:07.732873 8329 consensus_queue.cc:260] T 371146bfe9a940c98bd13143506a7322 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:07.733389 8329 ts_tablet_manager.cc:1398] T 371146bfe9a940c98bd13143506a7322 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:07.844796 7942 consensus_queue.cc:1035] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:07.849567 4704 raft_consensus.cc:3037] T d09a80f79473469e92310ded0f4edfdd P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:07.852051 7942 consensus_queue.cc:1035] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:07.861672 8168 raft_consensus.cc:482] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:07.861848 8168 raft_consensus.cc:504] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:07.862633 8168 leader_election.cc:288] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:07.863561 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371146bfe9a940c98bd13143506a7322" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:27:07.863898 4838 raft_consensus.cc:2454] T 371146bfe9a940c98bd13143506a7322 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:07.864500 4375 leader_election.cc:302] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:07.864876 8168 raft_consensus.cc:2781] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:07.865021 8168 raft_consensus.cc:482] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:07.865132 8168 raft_consensus.cc:3037] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:07.868161 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371146bfe9a940c98bd13143506a7322" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:07.868397 7484 raft_consensus.cc:2454] T 371146bfe9a940c98bd13143506a7322 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:07.869818 8168 raft_consensus.cc:504] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:07.870368 8168 leader_election.cc:288] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:07.870877 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371146bfe9a940c98bd13143506a7322" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:27:07.871112 4838 raft_consensus.cc:3037] T 371146bfe9a940c98bd13143506a7322 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:07.871121 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371146bfe9a940c98bd13143506a7322" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:07.871330 7484 raft_consensus.cc:3037] T 371146bfe9a940c98bd13143506a7322 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:07.874497 4838 raft_consensus.cc:2454] T 371146bfe9a940c98bd13143506a7322 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:07.875128 4375 leader_election.cc:302] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:07.875743 7484 raft_consensus.cc:2454] T 371146bfe9a940c98bd13143506a7322 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:07.876015 8335 raft_consensus.cc:2781] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:07.878338 8335 raft_consensus.cc:686] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:27:07.878661 8335 consensus_queue.cc:237] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:07.882119 8303 consensus_queue.cc:1035] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:07.881987 4300 catalog_manager.cc:5484] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:07.882611 8303 consensus_queue.cc:1035] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:07.942651 4300 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:07.950642 7330 tablet_service.cc:1467] Processing CreateTablet for tablet d714416dd87a49c6bd2d2b8de348c9df (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=8a6e0f9c1e92442cbf43f55d3eef080f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:07.950737 4686 tablet_service.cc:1467] Processing CreateTablet for tablet d714416dd87a49c6bd2d2b8de348c9df (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=8a6e0f9c1e92442cbf43f55d3eef080f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:07.951131 7330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d714416dd87a49c6bd2d2b8de348c9df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.951176 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d714416dd87a49c6bd2d2b8de348c9df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.951156 7595 tablet_service.cc:1467] Processing CreateTablet for tablet d714416dd87a49c6bd2d2b8de348c9df (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=8a6e0f9c1e92442cbf43f55d3eef080f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:07.951660 7595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d714416dd87a49c6bd2d2b8de348c9df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.956302 4421 tablet_service.cc:1467] Processing CreateTablet for tablet 030fa30553294e51808df57d6bb70b66 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=8a6e0f9c1e92442cbf43f55d3eef080f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:07.956790 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 030fa30553294e51808df57d6bb70b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.958667 7597 tablet_service.cc:1467] Processing CreateTablet for tablet b4298fc9f7694af19716ef4a908a501c (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=8a6e0f9c1e92442cbf43f55d3eef080f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:07.959184 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4298fc9f7694af19716ef4a908a501c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.960724 7194 tablet_service.cc:1467] Processing CreateTablet for tablet 030fa30553294e51808df57d6bb70b66 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=8a6e0f9c1e92442cbf43f55d3eef080f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:07.960758 7328 tablet_service.cc:1467] Processing CreateTablet for tablet 030fa30553294e51808df57d6bb70b66 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=8a6e0f9c1e92442cbf43f55d3eef080f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:07.961182 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 030fa30553294e51808df57d6bb70b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.961211 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 030fa30553294e51808df57d6bb70b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.961450 8343 tablet_bootstrap.cc:492] T d714416dd87a49c6bd2d2b8de348c9df P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:07.963143 8343 tablet_bootstrap.cc:654] T d714416dd87a49c6bd2d2b8de348c9df P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:07.963953 7329 tablet_service.cc:1467] Processing CreateTablet for tablet b4298fc9f7694af19716ef4a908a501c (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=8a6e0f9c1e92442cbf43f55d3eef080f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:07.964321 7329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4298fc9f7694af19716ef4a908a501c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.975924 8345 tablet_bootstrap.cc:492] T d714416dd87a49c6bd2d2b8de348c9df P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:07.975958 7462 tablet_service.cc:1467] Processing CreateTablet for tablet b4298fc9f7694af19716ef4a908a501c (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=8a6e0f9c1e92442cbf43f55d3eef080f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:07.976445 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4298fc9f7694af19716ef4a908a501c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.982772 8326 tablet_bootstrap.cc:492] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:07.984449 8326 tablet_bootstrap.cc:654] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:07.986018 8328 tablet_bootstrap.cc:492] T 030fa30553294e51808df57d6bb70b66 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:07.987754 8328 tablet_bootstrap.cc:654] T 030fa30553294e51808df57d6bb70b66 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:07.987973 8347 tablet_bootstrap.cc:492] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:07.989814 8326 tablet_bootstrap.cc:492] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:07.989991 8326 ts_tablet_manager.cc:1367] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 21:27:07.990216 8328 tablet_bootstrap.cc:492] T 030fa30553294e51808df57d6bb70b66 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:07.990363 8328 ts_tablet_manager.cc:1367] T 030fa30553294e51808df57d6bb70b66 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240502 21:27:07.990371 8345 tablet_bootstrap.cc:654] T d714416dd87a49c6bd2d2b8de348c9df P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:07.990796 8326 raft_consensus.cc:348] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:07.991039 8326 raft_consensus.cc:374] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:07.991113 8326 raft_consensus.cc:729] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:07.991360 8326 consensus_queue.cc:260] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:07.991954 8326 ts_tablet_manager.cc:1398] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:07.992014 8347 tablet_bootstrap.cc:654] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:07.994115 8328 raft_consensus.cc:348] T 030fa30553294e51808df57d6bb70b66 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:07.994398 8328 raft_consensus.cc:374] T 030fa30553294e51808df57d6bb70b66 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:07.994499 8328 raft_consensus.cc:729] T 030fa30553294e51808df57d6bb70b66 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:07.996212 8345 tablet_bootstrap.cc:492] T d714416dd87a49c6bd2d2b8de348c9df P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:07.996431 8345 ts_tablet_manager.cc:1367] T d714416dd87a49c6bd2d2b8de348c9df P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.003s
I20240502 21:27:07.997457 8345 raft_consensus.cc:348] T d714416dd87a49c6bd2d2b8de348c9df P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:07.997754 8345 raft_consensus.cc:374] T d714416dd87a49c6bd2d2b8de348c9df P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:07.997776 8343 tablet_bootstrap.cc:492] T d714416dd87a49c6bd2d2b8de348c9df P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:07.997937 8343 ts_tablet_manager.cc:1367] T d714416dd87a49c6bd2d2b8de348c9df P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.037s user 0.010s sys 0.005s
I20240502 21:27:07.997924 8345 raft_consensus.cc:729] T d714416dd87a49c6bd2d2b8de348c9df P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:07.998288 8345 consensus_queue.cc:260] T d714416dd87a49c6bd2d2b8de348c9df P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:07.998783 8343 raft_consensus.cc:348] T d714416dd87a49c6bd2d2b8de348c9df P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:07.998987 8343 raft_consensus.cc:374] T d714416dd87a49c6bd2d2b8de348c9df P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:07.999094 8343 raft_consensus.cc:729] T d714416dd87a49c6bd2d2b8de348c9df P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:07.999322 8343 consensus_queue.cc:260] T d714416dd87a49c6bd2d2b8de348c9df P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:08.002431 8343 ts_tablet_manager.cc:1398] T d714416dd87a49c6bd2d2b8de348c9df P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 21:27:08.002813 8343 tablet_bootstrap.cc:492] T b4298fc9f7694af19716ef4a908a501c P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:08.003926 8345 ts_tablet_manager.cc:1398] T d714416dd87a49c6bd2d2b8de348c9df P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 21:27:08.005499 8343 tablet_bootstrap.cc:654] T b4298fc9f7694af19716ef4a908a501c P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:08.007207 8345 tablet_bootstrap.cc:492] T 030fa30553294e51808df57d6bb70b66 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:08.007463 8347 tablet_bootstrap.cc:492] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:08.007813 8347 ts_tablet_manager.cc:1367] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240502 21:27:08.008981 8345 tablet_bootstrap.cc:654] T 030fa30553294e51808df57d6bb70b66 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:08.009752 8347 raft_consensus.cc:348] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:08.010393 8347 raft_consensus.cc:374] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.010574 8344 tablet_bootstrap.cc:492] T d714416dd87a49c6bd2d2b8de348c9df P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:08.010694 8347 raft_consensus.cc:729] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.011132 8325 raft_consensus.cc:482] T d714416dd87a49c6bd2d2b8de348c9df P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:08.011266 8325 raft_consensus.cc:504] T d714416dd87a49c6bd2d2b8de348c9df P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:08.011245 8347 consensus_queue.cc:260] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:08.012185 8325 leader_election.cc:288] T d714416dd87a49c6bd2d2b8de348c9df P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:07.994807 8328 consensus_queue.cc:260] T 030fa30553294e51808df57d6bb70b66 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:08.012701 8347 ts_tablet_manager.cc:1398] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:27:08.013106 8328 ts_tablet_manager.cc:1398] T 030fa30553294e51808df57d6bb70b66 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.023s user 0.000s sys 0.002s
I20240502 21:27:08.015738 8343 tablet_bootstrap.cc:492] T b4298fc9f7694af19716ef4a908a501c P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:08.015908 8343 ts_tablet_manager.cc:1367] T b4298fc9f7694af19716ef4a908a501c P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.005s
I20240502 21:27:08.016742 8343 raft_consensus.cc:348] T b4298fc9f7694af19716ef4a908a501c P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:08.017064 8343 raft_consensus.cc:374] T b4298fc9f7694af19716ef4a908a501c P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.017017 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d714416dd87a49c6bd2d2b8de348c9df" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:08.017140 8343 raft_consensus.cc:729] T b4298fc9f7694af19716ef4a908a501c P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.017227 7618 raft_consensus.cc:2454] T d714416dd87a49c6bd2d2b8de348c9df P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 0.
I20240502 21:27:08.017395 8343 consensus_queue.cc:260] T b4298fc9f7694af19716ef4a908a501c P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:08.017885 7282 leader_election.cc:302] T d714416dd87a49c6bd2d2b8de348c9df P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:08.018796 8343 ts_tablet_manager.cc:1398] T b4298fc9f7694af19716ef4a908a501c P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:08.021566 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:08.022045 8325 raft_consensus.cc:2781] T d714416dd87a49c6bd2d2b8de348c9df P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:08.022168 8325 raft_consensus.cc:482] T d714416dd87a49c6bd2d2b8de348c9df P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:08.022243 8325 raft_consensus.cc:3037] T d714416dd87a49c6bd2d2b8de348c9df P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:08.028573 8325 raft_consensus.cc:504] T d714416dd87a49c6bd2d2b8de348c9df P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:08.029287 8325 leader_election.cc:288] T d714416dd87a49c6bd2d2b8de348c9df P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:08.030019 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d714416dd87a49c6bd2d2b8de348c9df" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:08.030241 7618 raft_consensus.cc:3037] T d714416dd87a49c6bd2d2b8de348c9df P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:08.031883 4838 raft_consensus.cc:1261] T 371146bfe9a940c98bd13143506a7322 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:08.032786 8335 consensus_queue.cc:1035] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:08.032887 7618 raft_consensus.cc:2454] T d714416dd87a49c6bd2d2b8de348c9df P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
I20240502 21:27:08.033275 7282 leader_election.cc:302] T d714416dd87a49c6bd2d2b8de348c9df P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:08.034965 8325 raft_consensus.cc:2781] T d714416dd87a49c6bd2d2b8de348c9df P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:08.035514 8325 raft_consensus.cc:686] T d714416dd87a49c6bd2d2b8de348c9df P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Becoming Leader. State: Replica: d0456c5e55a542128688904e003d403c, State: Running, Role: LEADER
I20240502 21:27:08.037803 8325 consensus_queue.cc:237] T d714416dd87a49c6bd2d2b8de348c9df P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:08.040910 7484 raft_consensus.cc:1261] T 371146bfe9a940c98bd13143506a7322 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:08.041627 4300 catalog_manager.cc:5484] T d714416dd87a49c6bd2d2b8de348c9df P d0456c5e55a542128688904e003d403c reported cstate change: term changed from 0 to 1, leader changed from <none> to d0456c5e55a542128688904e003d403c (127.29.82.70). New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:08.042325 8335 consensus_queue.cc:1035] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:08.049088 8344 tablet_bootstrap.cc:654] T d714416dd87a49c6bd2d2b8de348c9df P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:08.055225 4300 catalog_manager.cc:5576] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449) does not have the latest schema for tablet 371146bfe9a940c98bd13143506a7322 (table rebalancer_dynamic_table [id=a9f1ef6bc55440d7a04bba730e02577e]). Expected version 1 got 0
I20240502 21:27:08.058642 8345 tablet_bootstrap.cc:492] T 030fa30553294e51808df57d6bb70b66 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:08.058843 8345 ts_tablet_manager.cc:1367] T 030fa30553294e51808df57d6bb70b66 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.003s
I20240502 21:27:08.059897 8345 raft_consensus.cc:348] T 030fa30553294e51808df57d6bb70b66 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:08.060191 8345 raft_consensus.cc:374] T 030fa30553294e51808df57d6bb70b66 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.060287 8345 raft_consensus.cc:729] T 030fa30553294e51808df57d6bb70b66 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.060577 8345 consensus_queue.cc:260] T 030fa30553294e51808df57d6bb70b66 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:08.061192 8345 ts_tablet_manager.cc:1398] T 030fa30553294e51808df57d6bb70b66 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:08.061403 4298 catalog_manager.cc:5576] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449) does not have the latest schema for tablet 371146bfe9a940c98bd13143506a7322 (table rebalancer_dynamic_table [id=a9f1ef6bc55440d7a04bba730e02577e]). Expected version 1 got 0
I20240502 21:27:08.061590 8345 tablet_bootstrap.cc:492] T b4298fc9f7694af19716ef4a908a501c P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:08.061705 8364 tablet.cc:1722] T 371146bfe9a940c98bd13143506a7322 P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:08.064189 8345 tablet_bootstrap.cc:654] T b4298fc9f7694af19716ef4a908a501c P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:08.064783 8344 tablet_bootstrap.cc:492] T d714416dd87a49c6bd2d2b8de348c9df P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:27:08.065006 8344 ts_tablet_manager.cc:1367] T d714416dd87a49c6bd2d2b8de348c9df P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.000s
I20240502 21:27:08.066391 8344 raft_consensus.cc:348] T d714416dd87a49c6bd2d2b8de348c9df P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:08.066553 8303 raft_consensus.cc:482] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:08.066704 8344 raft_consensus.cc:374] T d714416dd87a49c6bd2d2b8de348c9df P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.066745 8303 raft_consensus.cc:504] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:08.066864 8344 raft_consensus.cc:729] T d714416dd87a49c6bd2d2b8de348c9df P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.067207 8344 consensus_queue.cc:260] T d714416dd87a49c6bd2d2b8de348c9df P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:08.067492 8303 leader_election.cc:288] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:08.067628 4300 catalog_manager.cc:5576] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913) does not have the latest schema for tablet 371146bfe9a940c98bd13143506a7322 (table rebalancer_dynamic_table [id=a9f1ef6bc55440d7a04bba730e02577e]). Expected version 1 got 0
I20240502 21:27:08.068516 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4298fc9f7694af19716ef4a908a501c" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:08.069595 8344 ts_tablet_manager.cc:1398] T d714416dd87a49c6bd2d2b8de348c9df P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:08.071832 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4298fc9f7694af19716ef4a908a501c" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:08.072391 7618 raft_consensus.cc:2454] T b4298fc9f7694af19716ef4a908a501c P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
W20240502 21:27:08.073419 7419 leader_election.cc:341] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:08.073730 7418 leader_election.cc:302] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168; no voters: d0456c5e55a542128688904e003d403c
I20240502 21:27:08.074268 8341 raft_consensus.cc:2781] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:08.074369 8341 raft_consensus.cc:482] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:08.074465 8341 raft_consensus.cc:3037] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:08.074810 8363 tablet.cc:1722] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:08.080468 8363 tablet.cc:1620] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:08.089232 8345 tablet_bootstrap.cc:492] T b4298fc9f7694af19716ef4a908a501c P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:08.089231 8341 raft_consensus.cc:504] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:08.089458 8345 ts_tablet_manager.cc:1367] T b4298fc9f7694af19716ef4a908a501c P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.012s
I20240502 21:27:08.092087 8345 raft_consensus.cc:348] T b4298fc9f7694af19716ef4a908a501c P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:08.092370 8345 raft_consensus.cc:374] T b4298fc9f7694af19716ef4a908a501c P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.092474 8345 raft_consensus.cc:729] T b4298fc9f7694af19716ef4a908a501c P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.092763 8345 consensus_queue.cc:260] T b4298fc9f7694af19716ef4a908a501c P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:08.093509 8345 ts_tablet_manager.cc:1398] T b4298fc9f7694af19716ef4a908a501c P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 21:27:08.096427 8341 leader_election.cc:288] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:08.096719 4704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d714416dd87a49c6bd2d2b8de348c9df" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:27:08.096954 4704 raft_consensus.cc:3037] T d714416dd87a49c6bd2d2b8de348c9df P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:08.097813 4298 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=a9f1ef6bc55440d7a04bba730e02577e] alter complete (version 1)
I20240502 21:27:08.098565 8364 tablet.cc:1620] T 371146bfe9a940c98bd13143506a7322 P 73a1f2ca1f544de9b0e3b37015f85328: MemRowSet was empty: no flush needed.
I20240502 21:27:08.099422 4704 raft_consensus.cc:2454] T d714416dd87a49c6bd2d2b8de348c9df P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
I20240502 21:27:08.099603 8366 tablet.cc:1722] T 371146bfe9a940c98bd13143506a7322 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:08.100001 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d714416dd87a49c6bd2d2b8de348c9df" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:27:08.100250 8366 tablet.cc:1620] T 371146bfe9a940c98bd13143506a7322 P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:08.100217 4705 raft_consensus.cc:2362] T d714416dd87a49c6bd2d2b8de348c9df P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d0456c5e55a542128688904e003d403c in term 1. Re-sending same reply.
I20240502 21:27:08.101866 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4298fc9f7694af19716ef4a908a501c" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:08.102056 7350 raft_consensus.cc:3037] T b4298fc9f7694af19716ef4a908a501c P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:08.104786 7350 raft_consensus.cc:2454] T b4298fc9f7694af19716ef4a908a501c P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:08.104976 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4298fc9f7694af19716ef4a908a501c" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:08.105213 7618 raft_consensus.cc:3037] T b4298fc9f7694af19716ef4a908a501c P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:08.105741 4300 catalog_manager.cc:5576] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913) does not have the latest schema for tablet 371146bfe9a940c98bd13143506a7322 (table rebalancer_dynamic_table [id=a9f1ef6bc55440d7a04bba730e02577e]). Expected version 1 got 0
I20240502 21:27:08.106247 7419 leader_election.cc:302] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:08.107765 8341 raft_consensus.cc:2781] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:08.108009 8341 raft_consensus.cc:686] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Becoming Leader. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Running, Role: LEADER
I20240502 21:27:08.108009 7618 raft_consensus.cc:2454] T b4298fc9f7694af19716ef4a908a501c P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:08.108289 8341 consensus_queue.cc:237] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:08.113122 4298 catalog_manager.cc:5484] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a72286397c4edc9a63eca3412be168 (127.29.82.71). New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:08.202186 4298 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:08.209803 8363 tablet.cc:1722] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:08.210340 8363 tablet.cc:1620] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:08.211236 8364 tablet.cc:1722] T 371146bfe9a940c98bd13143506a7322 P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:08.211684 8366 tablet.cc:1722] T 371146bfe9a940c98bd13143506a7322 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:08.212131 8366 tablet.cc:1620] T 371146bfe9a940c98bd13143506a7322 P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:08.212874 8364 tablet.cc:1620] T 371146bfe9a940c98bd13143506a7322 P 73a1f2ca1f544de9b0e3b37015f85328: MemRowSet was empty: no flush needed.
I20240502 21:27:08.221146 4298 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=a9f1ef6bc55440d7a04bba730e02577e] alter complete (version 2)
I20240502 21:27:08.249007 7970 raft_consensus.cc:482] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:08.249239 7970 raft_consensus.cc:504] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:08.250043 7970 leader_election.cc:288] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:08.250711 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "030fa30553294e51808df57d6bb70b66" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:27:08.250988 4443 raft_consensus.cc:2454] T 030fa30553294e51808df57d6bb70b66 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:08.251644 7148 leader_election.cc:302] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:08.252501 7970 raft_consensus.cc:2781] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:08.252605 7970 raft_consensus.cc:482] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:08.252728 7970 raft_consensus.cc:3037] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:08.252640 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "030fa30553294e51808df57d6bb70b66" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:08.253240 7350 raft_consensus.cc:2454] T 030fa30553294e51808df57d6bb70b66 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:08.255282 7970 raft_consensus.cc:504] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:08.256445 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "030fa30553294e51808df57d6bb70b66" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:08.256546 7970 leader_election.cc:288] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:08.256739 7350 raft_consensus.cc:3037] T 030fa30553294e51808df57d6bb70b66 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:08.256806 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "030fa30553294e51808df57d6bb70b66" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:27:08.256987 4443 raft_consensus.cc:3037] T 030fa30553294e51808df57d6bb70b66 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:08.258863 7350 raft_consensus.cc:2454] T 030fa30553294e51808df57d6bb70b66 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:08.259001 4443 raft_consensus.cc:2454] T 030fa30553294e51808df57d6bb70b66 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:08.259279 7150 leader_election.cc:302] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:08.259622 7970 raft_consensus.cc:2781] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:08.259858 7970 raft_consensus.cc:686] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:27:08.260177 7970 consensus_queue.cc:237] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:08.263396 4294 catalog_manager.cc:5484] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69). New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:08.323997 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:08.332811 8363 tablet.cc:1722] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:08.333256 8364 tablet.cc:1722] T 371146bfe9a940c98bd13143506a7322 P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:08.333860 8366 tablet.cc:1722] T 371146bfe9a940c98bd13143506a7322 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:08.343804 4294 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=a9f1ef6bc55440d7a04bba730e02577e] alter complete (version 3)
I20240502 21:27:08.431128 8325 consensus_queue.cc:1035] T d714416dd87a49c6bd2d2b8de348c9df P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:08.436889 8325 consensus_queue.cc:1035] T d714416dd87a49c6bd2d2b8de348c9df P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:08.448840 4298 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:08.449049 4298 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:08.451821 4298 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 371146bfe9a940c98bd13143506a7322
I20240502 21:27:08.452651 7462 tablet_service.cc:1514] Processing DeleteTablet for tablet 371146bfe9a940c98bd13143506a7322 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:08 UTC) from {username='slave'} at 127.0.0.1:37182
I20240502 21:27:08.452684 4818 tablet_service.cc:1514] Processing DeleteTablet for tablet 371146bfe9a940c98bd13143506a7322 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:08 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:27:08.452656 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet 371146bfe9a940c98bd13143506a7322 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:08 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:08.453418 8376 tablet_replica.cc:330] T 371146bfe9a940c98bd13143506a7322 P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:27:08.453595 8375 tablet_replica.cc:330] T 371146bfe9a940c98bd13143506a7322 P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:27:08.453747 8376 raft_consensus.cc:2229] T 371146bfe9a940c98bd13143506a7322 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:08.453783 8377 tablet_replica.cc:330] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:08.453840 8375 raft_consensus.cc:2229] T 371146bfe9a940c98bd13143506a7322 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:08.453972 8376 raft_consensus.cc:2258] T 371146bfe9a940c98bd13143506a7322 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:08.453987 8377 raft_consensus.cc:2229] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:08.454078 8375 raft_consensus.cc:2258] T 371146bfe9a940c98bd13143506a7322 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:08.454401 8377 raft_consensus.cc:2258] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:08.454941 8376 ts_tablet_manager.cc:1875] T 371146bfe9a940c98bd13143506a7322 P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:08.455009 8375 ts_tablet_manager.cc:1875] T 371146bfe9a940c98bd13143506a7322 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:08.455216 8377 ts_tablet_manager.cc:1875] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:08.459832 8376 ts_tablet_manager.cc:1888] T 371146bfe9a940c98bd13143506a7322 P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:08.459951 8376 log.cc:1196] T 371146bfe9a940c98bd13143506a7322 P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/371146bfe9a940c98bd13143506a7322
I20240502 21:27:08.460414 8376 ts_tablet_manager.cc:1909] T 371146bfe9a940c98bd13143506a7322 P 31a72286397c4edc9a63eca3412be168: Deleting consensus metadata
I20240502 21:27:08.461365 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet 371146bfe9a940c98bd13143506a7322 (table rebalancer_dynamic_table_new_name [id=a9f1ef6bc55440d7a04bba730e02577e]) successfully deleted
I20240502 21:27:08.461549 8377 ts_tablet_manager.cc:1888] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:08.461549 8375 ts_tablet_manager.cc:1888] T 371146bfe9a940c98bd13143506a7322 P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:08.461680 8377 log.cc:1196] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/371146bfe9a940c98bd13143506a7322
I20240502 21:27:08.461680 8375 log.cc:1196] T 371146bfe9a940c98bd13143506a7322 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/371146bfe9a940c98bd13143506a7322
I20240502 21:27:08.462177 8375 ts_tablet_manager.cc:1909] T 371146bfe9a940c98bd13143506a7322 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:27:08.462165 8377 ts_tablet_manager.cc:1909] T 371146bfe9a940c98bd13143506a7322 P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:08.463214 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet 371146bfe9a940c98bd13143506a7322 (table rebalancer_dynamic_table_new_name [id=a9f1ef6bc55440d7a04bba730e02577e]) successfully deleted
I20240502 21:27:08.463397 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet 371146bfe9a940c98bd13143506a7322 (table rebalancer_dynamic_table_new_name [id=a9f1ef6bc55440d7a04bba730e02577e]) successfully deleted
I20240502 21:27:08.554085 4298 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:08.561717 4818 tablet_service.cc:1467] Processing CreateTablet for tablet a0781e3a014a483d91a523e7cf1434b7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=08a1406d2f624b54be9892ab8b1b7b3b]), partition=
I20240502 21:27:08.561761 7194 tablet_service.cc:1467] Processing CreateTablet for tablet a0781e3a014a483d91a523e7cf1434b7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=08a1406d2f624b54be9892ab8b1b7b3b]), partition=
I20240502 21:27:08.562122 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0781e3a014a483d91a523e7cf1434b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:08.562124 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0781e3a014a483d91a523e7cf1434b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:08.566826 8378 tablet_bootstrap.cc:492] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:08.567011 7329 tablet_service.cc:1467] Processing CreateTablet for tablet a0781e3a014a483d91a523e7cf1434b7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=08a1406d2f624b54be9892ab8b1b7b3b]), partition=
I20240502 21:27:08.567389 7329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0781e3a014a483d91a523e7cf1434b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:08.567409 8379 tablet_bootstrap.cc:492] T a0781e3a014a483d91a523e7cf1434b7 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:08.568718 8378 tablet_bootstrap.cc:654] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:08.569046 8379 tablet_bootstrap.cc:654] T a0781e3a014a483d91a523e7cf1434b7 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:08.571239 8378 tablet_bootstrap.cc:492] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:08.571384 8378 ts_tablet_manager.cc:1367] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:27:08.571702 8379 tablet_bootstrap.cc:492] T a0781e3a014a483d91a523e7cf1434b7 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:08.571839 8379 ts_tablet_manager.cc:1367] T a0781e3a014a483d91a523e7cf1434b7 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 21:27:08.572156 8378 raft_consensus.cc:348] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:08.572407 8378 raft_consensus.cc:374] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.572448 8379 raft_consensus.cc:348] T a0781e3a014a483d91a523e7cf1434b7 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:08.572541 8378 raft_consensus.cc:729] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.572659 8379 raft_consensus.cc:374] T a0781e3a014a483d91a523e7cf1434b7 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.572803 8379 raft_consensus.cc:729] T a0781e3a014a483d91a523e7cf1434b7 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.572780 8378 consensus_queue.cc:260] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:08.573035 8379 consensus_queue.cc:260] T a0781e3a014a483d91a523e7cf1434b7 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:08.573671 8379 ts_tablet_manager.cc:1398] T a0781e3a014a483d91a523e7cf1434b7 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:08.573647 8378 ts_tablet_manager.cc:1398] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 21:27:08.574853 8345 tablet_bootstrap.cc:492] T a0781e3a014a483d91a523e7cf1434b7 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:08.576196 8345 tablet_bootstrap.cc:654] T a0781e3a014a483d91a523e7cf1434b7 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:08.579799 8345 tablet_bootstrap.cc:492] T a0781e3a014a483d91a523e7cf1434b7 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:08.579975 8345 ts_tablet_manager.cc:1367] T a0781e3a014a483d91a523e7cf1434b7 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:27:08.580757 8345 raft_consensus.cc:348] T a0781e3a014a483d91a523e7cf1434b7 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:08.580964 8345 raft_consensus.cc:374] T a0781e3a014a483d91a523e7cf1434b7 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.581063 8345 raft_consensus.cc:729] T a0781e3a014a483d91a523e7cf1434b7 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.581331 8345 consensus_queue.cc:260] T a0781e3a014a483d91a523e7cf1434b7 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:08.581854 8345 ts_tablet_manager.cc:1398] T a0781e3a014a483d91a523e7cf1434b7 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:08.581751 8341 consensus_queue.cc:1035] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:08.588269 8303 consensus_queue.cc:1035] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:08.645364 7970 consensus_queue.cc:1035] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:08.650527 8339 consensus_queue.cc:1035] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:08.788859 4298 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:08.797994 4421 tablet_service.cc:1467] Processing CreateTablet for tablet 42edb2b3ff094c71872fde7949b61b3b (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=50e84343189b44c5bd6158fbd7ba5394]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:08.798470 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42edb2b3ff094c71872fde7949b61b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:08.798478 4555 tablet_service.cc:1467] Processing CreateTablet for tablet 42edb2b3ff094c71872fde7949b61b3b (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=50e84343189b44c5bd6158fbd7ba5394]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:08.798921 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42edb2b3ff094c71872fde7949b61b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:08.803656 4553 tablet_service.cc:1467] Processing CreateTablet for tablet 2eb73e73b64b4b0f9e5116f258f2e147 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=50e84343189b44c5bd6158fbd7ba5394]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:08.804046 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2eb73e73b64b4b0f9e5116f258f2e147. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:08.804261 4422 tablet_service.cc:1467] Processing CreateTablet for tablet 2eb73e73b64b4b0f9e5116f258f2e147 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=50e84343189b44c5bd6158fbd7ba5394]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:08.804518 4423 tablet_service.cc:1467] Processing CreateTablet for tablet ca332df7aaef4ddd9fd3266e8003b55f (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=50e84343189b44c5bd6158fbd7ba5394]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:08.804630 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2eb73e73b64b4b0f9e5116f258f2e147. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:08.804848 4423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca332df7aaef4ddd9fd3266e8003b55f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:08.805450 7195 tablet_service.cc:1467] Processing CreateTablet for tablet ca332df7aaef4ddd9fd3266e8003b55f (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=50e84343189b44c5bd6158fbd7ba5394]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:08.805501 7194 tablet_service.cc:1467] Processing CreateTablet for tablet 42edb2b3ff094c71872fde7949b61b3b (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=50e84343189b44c5bd6158fbd7ba5394]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:08.805902 7195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca332df7aaef4ddd9fd3266e8003b55f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:08.807996 7193 tablet_service.cc:1467] Processing CreateTablet for tablet 2eb73e73b64b4b0f9e5116f258f2e147 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=50e84343189b44c5bd6158fbd7ba5394]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:08.808601 7329 tablet_service.cc:1467] Processing CreateTablet for tablet ca332df7aaef4ddd9fd3266e8003b55f (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=50e84343189b44c5bd6158fbd7ba5394]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:08.809027 7329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca332df7aaef4ddd9fd3266e8003b55f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:08.811939 7193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2eb73e73b64b4b0f9e5116f258f2e147. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:08.814630 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42edb2b3ff094c71872fde7949b61b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:08.815469 8389 tablet_bootstrap.cc:492] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:08.818125 8389 tablet_bootstrap.cc:654] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:08.829409 8389 tablet_bootstrap.cc:492] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:08.829739 8389 ts_tablet_manager.cc:1367] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20240502 21:27:08.830734 8389 raft_consensus.cc:348] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:08.831112 8389 raft_consensus.cc:374] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.831296 8389 raft_consensus.cc:729] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.831665 8389 consensus_queue.cc:260] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:08.836480 8389 ts_tablet_manager.cc:1398] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 21:27:08.850409 8379 tablet_bootstrap.cc:492] T ca332df7aaef4ddd9fd3266e8003b55f P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:08.852252 8379 tablet_bootstrap.cc:654] T ca332df7aaef4ddd9fd3266e8003b55f P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:08.855923 8389 tablet_bootstrap.cc:492] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:08.857596 8389 tablet_bootstrap.cc:654] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:08.859947 8345 tablet_bootstrap.cc:492] T ca332df7aaef4ddd9fd3266e8003b55f P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:08.860213 8379 tablet_bootstrap.cc:492] T ca332df7aaef4ddd9fd3266e8003b55f P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:08.860486 8379 ts_tablet_manager.cc:1367] T ca332df7aaef4ddd9fd3266e8003b55f P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240502 21:27:08.861276 8388 tablet_bootstrap.cc:492] T 42edb2b3ff094c71872fde7949b61b3b P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:08.861375 8379 raft_consensus.cc:348] T ca332df7aaef4ddd9fd3266e8003b55f P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:08.861676 8345 tablet_bootstrap.cc:654] T ca332df7aaef4ddd9fd3266e8003b55f P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:08.861789 8379 raft_consensus.cc:374] T ca332df7aaef4ddd9fd3266e8003b55f P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.861927 8379 raft_consensus.cc:729] T ca332df7aaef4ddd9fd3266e8003b55f P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.862262 8379 consensus_queue.cc:260] T ca332df7aaef4ddd9fd3266e8003b55f P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:08.863032 8388 tablet_bootstrap.cc:654] T 42edb2b3ff094c71872fde7949b61b3b P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:08.864336 8379 ts_tablet_manager.cc:1398] T ca332df7aaef4ddd9fd3266e8003b55f P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:08.864708 8379 tablet_bootstrap.cc:492] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:08.868669 8379 tablet_bootstrap.cc:654] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:08.875993 8389 tablet_bootstrap.cc:492] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:08.876169 8389 ts_tablet_manager.cc:1367] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.001s
I20240502 21:27:08.876639 8379 tablet_bootstrap.cc:492] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:08.876778 8379 ts_tablet_manager.cc:1367] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.001s
I20240502 21:27:08.877125 8345 tablet_bootstrap.cc:492] T ca332df7aaef4ddd9fd3266e8003b55f P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:08.877274 8345 ts_tablet_manager.cc:1367] T ca332df7aaef4ddd9fd3266e8003b55f P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.006s
I20240502 21:27:08.877590 8379 raft_consensus.cc:348] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:08.877786 8379 raft_consensus.cc:374] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.877893 8379 raft_consensus.cc:729] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.878175 8379 consensus_queue.cc:260] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:08.878144 8345 raft_consensus.cc:348] T ca332df7aaef4ddd9fd3266e8003b55f P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:08.878363 8345 raft_consensus.cc:374] T ca332df7aaef4ddd9fd3266e8003b55f P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.878458 8345 raft_consensus.cc:729] T ca332df7aaef4ddd9fd3266e8003b55f P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.876907 8389 raft_consensus.cc:348] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:08.878772 8389 raft_consensus.cc:374] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.878852 8389 raft_consensus.cc:729] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.878749 8345 consensus_queue.cc:260] T ca332df7aaef4ddd9fd3266e8003b55f P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:08.879077 8389 consensus_queue.cc:260] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:08.879755 8379 ts_tablet_manager.cc:1398] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:27:08.880034 8379 tablet_bootstrap.cc:492] T 2eb73e73b64b4b0f9e5116f258f2e147 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:08.880790 8389 ts_tablet_manager.cc:1398] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 21:27:08.881101 8389 tablet_bootstrap.cc:492] T 42edb2b3ff094c71872fde7949b61b3b P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:08.882580 8389 tablet_bootstrap.cc:654] T 42edb2b3ff094c71872fde7949b61b3b P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:08.883942 8345 ts_tablet_manager.cc:1398] T ca332df7aaef4ddd9fd3266e8003b55f P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 21:27:08.890808 8379 tablet_bootstrap.cc:654] T 2eb73e73b64b4b0f9e5116f258f2e147 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:08.893066 8388 tablet_bootstrap.cc:492] T 42edb2b3ff094c71872fde7949b61b3b P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:08.893316 8388 ts_tablet_manager.cc:1367] T 42edb2b3ff094c71872fde7949b61b3b P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.011s
I20240502 21:27:08.893651 8389 tablet_bootstrap.cc:492] T 42edb2b3ff094c71872fde7949b61b3b P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:08.893857 8389 ts_tablet_manager.cc:1367] T 42edb2b3ff094c71872fde7949b61b3b P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.004s
I20240502 21:27:08.894392 8388 raft_consensus.cc:348] T 42edb2b3ff094c71872fde7949b61b3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:08.894637 8379 tablet_bootstrap.cc:492] T 2eb73e73b64b4b0f9e5116f258f2e147 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:08.894605 8389 raft_consensus.cc:348] T 42edb2b3ff094c71872fde7949b61b3b P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:08.894748 8388 raft_consensus.cc:374] T 42edb2b3ff094c71872fde7949b61b3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.894812 8379 ts_tablet_manager.cc:1367] T 2eb73e73b64b4b0f9e5116f258f2e147 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.003s
I20240502 21:27:08.894896 8389 raft_consensus.cc:374] T 42edb2b3ff094c71872fde7949b61b3b P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.894915 8388 raft_consensus.cc:729] T 42edb2b3ff094c71872fde7949b61b3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.895026 8389 raft_consensus.cc:729] T 42edb2b3ff094c71872fde7949b61b3b P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.895308 8389 consensus_queue.cc:260] T 42edb2b3ff094c71872fde7949b61b3b P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:08.895290 8388 consensus_queue.cc:260] T 42edb2b3ff094c71872fde7949b61b3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:08.895990 8389 ts_tablet_manager.cc:1398] T 42edb2b3ff094c71872fde7949b61b3b P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:27:08.896555 8388 ts_tablet_manager.cc:1398] T 42edb2b3ff094c71872fde7949b61b3b P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:27:08.896937 8379 raft_consensus.cc:348] T 2eb73e73b64b4b0f9e5116f258f2e147 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:08.897161 8379 raft_consensus.cc:374] T 2eb73e73b64b4b0f9e5116f258f2e147 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.897229 8379 raft_consensus.cc:729] T 2eb73e73b64b4b0f9e5116f258f2e147 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.897315 8388 tablet_bootstrap.cc:492] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:08.897447 8379 consensus_queue.cc:260] T 2eb73e73b64b4b0f9e5116f258f2e147 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:08.898746 8379 ts_tablet_manager.cc:1398] T 2eb73e73b64b4b0f9e5116f258f2e147 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 21:27:08.899552 8388 tablet_bootstrap.cc:654] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:08.907083 8388 tablet_bootstrap.cc:492] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:08.907217 8388 ts_tablet_manager.cc:1367] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20240502 21:27:08.907847 8388 raft_consensus.cc:348] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:08.908077 8388 raft_consensus.cc:374] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.908195 8388 raft_consensus.cc:729] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.908449 8388 consensus_queue.cc:260] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:08.908924 8388 ts_tablet_manager.cc:1398] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:08.940894 8339 raft_consensus.cc:482] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:08.941133 8339 raft_consensus.cc:504] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:08.941908 8339 leader_election.cc:288] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:27:08.942442 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42edb2b3ff094c71872fde7949b61b3b" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:27:08.942710 4443 raft_consensus.cc:2454] T 42edb2b3ff094c71872fde7949b61b3b P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:08.943152 7148 leader_election.cc:302] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:08.943241 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42edb2b3ff094c71872fde7949b61b3b" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:27:08.943356 8270 raft_consensus.cc:482] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:08.943461 8339 raft_consensus.cc:2781] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:08.943488 4574 raft_consensus.cc:2454] T 42edb2b3ff094c71872fde7949b61b3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:08.943549 8339 raft_consensus.cc:482] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:08.943683 8339 raft_consensus.cc:3037] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:08.943522 8270 raft_consensus.cc:504] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:08.944324 8270 leader_election.cc:288] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:08.944893 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0781e3a014a483d91a523e7cf1434b7" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:08.945154 7215 raft_consensus.cc:2454] T a0781e3a014a483d91a523e7cf1434b7 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:27:08.945133 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0781e3a014a483d91a523e7cf1434b7" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:08.945369 7350 raft_consensus.cc:2454] T a0781e3a014a483d91a523e7cf1434b7 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:27:08.945684 4772 leader_election.cc:302] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:08.945955 8270 raft_consensus.cc:2781] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:08.946074 8270 raft_consensus.cc:482] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:08.946179 8270 raft_consensus.cc:3037] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:08.946590 8339 raft_consensus.cc:504] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:08.947157 8339 leader_election.cc:288] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:27:08.947541 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42edb2b3ff094c71872fde7949b61b3b" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:27:08.947893 4443 raft_consensus.cc:3037] T 42edb2b3ff094c71872fde7949b61b3b P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:08.948838 8270 raft_consensus.cc:504] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:08.949079 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42edb2b3ff094c71872fde7949b61b3b" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:27:08.949308 4574 raft_consensus.cc:3037] T 42edb2b3ff094c71872fde7949b61b3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:08.949471 8270 leader_election.cc:288] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:08.950203 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0781e3a014a483d91a523e7cf1434b7" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:08.950242 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0781e3a014a483d91a523e7cf1434b7" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:08.950417 7215 raft_consensus.cc:3037] T a0781e3a014a483d91a523e7cf1434b7 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:08.950435 7350 raft_consensus.cc:3037] T a0781e3a014a483d91a523e7cf1434b7 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:08.951701 4574 raft_consensus.cc:2454] T 42edb2b3ff094c71872fde7949b61b3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:08.952235 7151 leader_election.cc:302] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:27:08.952625 8339 raft_consensus.cc:2781] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:08.952765 4443 raft_consensus.cc:2454] T 42edb2b3ff094c71872fde7949b61b3b P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:08.952996 8339 raft_consensus.cc:686] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:27:08.953162 7350 raft_consensus.cc:2454] T a0781e3a014a483d91a523e7cf1434b7 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:27:08.953166 7215 raft_consensus.cc:2454] T a0781e3a014a483d91a523e7cf1434b7 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:27:08.953560 4772 leader_election.cc:302] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:08.953378 8339 consensus_queue.cc:237] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:08.953908 8270 raft_consensus.cc:2781] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:08.954164 8270 raft_consensus.cc:686] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:27:08.954964 8270 consensus_queue.cc:237] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:08.957587 4298 catalog_manager.cc:5484] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69). New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:08.958941 4291 catalog_manager.cc:5484] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:09.022794 8168 raft_consensus.cc:482] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:09.023000 8168 raft_consensus.cc:504] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:09.023878 8168 leader_election.cc:288] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:27:09.024441 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eb73e73b64b4b0f9e5116f258f2e147" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:09.024741 7215 raft_consensus.cc:2454] T 2eb73e73b64b4b0f9e5116f258f2e147 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:09.024722 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eb73e73b64b4b0f9e5116f258f2e147" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:27:09.025033 4574 raft_consensus.cc:2454] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:09.025354 4377 leader_election.cc:302] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:09.025676 8168 raft_consensus.cc:2781] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:09.025794 8168 raft_consensus.cc:482] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:09.025892 8168 raft_consensus.cc:3037] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:09.028223 8168 raft_consensus.cc:504] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:09.029351 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eb73e73b64b4b0f9e5116f258f2e147" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:09.029467 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eb73e73b64b4b0f9e5116f258f2e147" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:27:09.029564 7215 raft_consensus.cc:3037] T 2eb73e73b64b4b0f9e5116f258f2e147 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:09.029747 4574 raft_consensus.cc:3037] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:09.031973 7215 raft_consensus.cc:2454] T 2eb73e73b64b4b0f9e5116f258f2e147 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:09.028931 8168 leader_election.cc:288] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:27:09.032426 4574 raft_consensus.cc:2454] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:09.033270 4378 leader_election.cc:302] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:27:09.033977 8168 raft_consensus.cc:2781] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:09.034265 8168 raft_consensus.cc:686] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:27:09.034752 8168 consensus_queue.cc:237] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:09.039482 4291 catalog_manager.cc:5484] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:09.129127 4291 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:09.135303 7350 raft_consensus.cc:1261] T a0781e3a014a483d91a523e7cf1434b7 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:09.135339 7215 raft_consensus.cc:1261] T a0781e3a014a483d91a523e7cf1434b7 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:09.136054 8270 consensus_queue.cc:1035] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:09.136632 8270 consensus_queue.cc:1035] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:09.141752 4291 catalog_manager.cc:5576] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719) does not have the latest schema for tablet a0781e3a014a483d91a523e7cf1434b7 (table rebalancer_dynamic_table [id=08a1406d2f624b54be9892ab8b1b7b3b]). Expected version 1 got 0
I20240502 21:27:09.142818 8408 tablet.cc:1722] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:09.143807 8408 tablet.cc:1620] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328: MemRowSet was empty: no flush needed.
I20240502 21:27:09.145706 8409 tablet.cc:1722] T a0781e3a014a483d91a523e7cf1434b7 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:09.146167 4291 catalog_manager.cc:5576] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193) does not have the latest schema for tablet a0781e3a014a483d91a523e7cf1434b7 (table rebalancer_dynamic_table [id=08a1406d2f624b54be9892ab8b1b7b3b]). Expected version 1 got 0
I20240502 21:27:09.146448 8409 tablet.cc:1620] T a0781e3a014a483d91a523e7cf1434b7 P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:09.147194 8411 tablet.cc:1722] T a0781e3a014a483d91a523e7cf1434b7 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:09.147897 8411 tablet.cc:1620] T a0781e3a014a483d91a523e7cf1434b7 P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:09.149051 4291 catalog_manager.cc:5576] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719) does not have the latest schema for tablet a0781e3a014a483d91a523e7cf1434b7 (table rebalancer_dynamic_table [id=08a1406d2f624b54be9892ab8b1b7b3b]). Expected version 1 got 0
I20240502 21:27:09.152953 4298 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=08a1406d2f624b54be9892ab8b1b7b3b] alter complete (version 1)
I20240502 21:27:09.255246 4291 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:09.263206 8408 tablet.cc:1722] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:09.263965 8408 tablet.cc:1620] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328: MemRowSet was empty: no flush needed.
I20240502 21:27:09.264253 8403 raft_consensus.cc:482] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:09.264427 8403 raft_consensus.cc:504] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:09.264606 8411 tablet.cc:1722] T a0781e3a014a483d91a523e7cf1434b7 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:09.265223 8403 leader_election.cc:288] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:09.265635 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca332df7aaef4ddd9fd3266e8003b55f" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:09.265863 7215 raft_consensus.cc:2454] T ca332df7aaef4ddd9fd3266e8003b55f P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:09.266063 8411 tablet.cc:1620] T a0781e3a014a483d91a523e7cf1434b7 P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:09.265937 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca332df7aaef4ddd9fd3266e8003b55f" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:09.266227 7350 raft_consensus.cc:2454] T ca332df7aaef4ddd9fd3266e8003b55f P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:09.266305 4377 leader_election.cc:302] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:09.266624 8403 raft_consensus.cc:2781] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:09.266744 8403 raft_consensus.cc:482] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:09.266831 8403 raft_consensus.cc:3037] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:09.266846 8409 tablet.cc:1722] T a0781e3a014a483d91a523e7cf1434b7 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:09.269974 8403 raft_consensus.cc:504] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:09.270491 8409 tablet.cc:1620] T a0781e3a014a483d91a523e7cf1434b7 P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:09.272248 8403 leader_election.cc:288] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:09.272624 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca332df7aaef4ddd9fd3266e8003b55f" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:09.272830 7215 raft_consensus.cc:3037] T ca332df7aaef4ddd9fd3266e8003b55f P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:09.273456 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca332df7aaef4ddd9fd3266e8003b55f" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:09.273641 7350 raft_consensus.cc:3037] T ca332df7aaef4ddd9fd3266e8003b55f P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:09.275534 7215 raft_consensus.cc:2454] T ca332df7aaef4ddd9fd3266e8003b55f P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:09.275555 7350 raft_consensus.cc:2454] T ca332df7aaef4ddd9fd3266e8003b55f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:09.276026 4377 leader_election.cc:302] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:09.276624 8403 raft_consensus.cc:2781] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:09.276778 4291 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=08a1406d2f624b54be9892ab8b1b7b3b] alter complete (version 2)
I20240502 21:27:09.277135 8403 raft_consensus.cc:686] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:27:09.277455 8403 consensus_queue.cc:237] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:09.283481 4298 catalog_manager.cc:5484] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:09.378746 4298 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:09.386746 8408 tablet.cc:1722] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:09.387470 8411 tablet.cc:1722] T a0781e3a014a483d91a523e7cf1434b7 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:09.390082 8409 tablet.cc:1722] T a0781e3a014a483d91a523e7cf1434b7 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:09.396116 4298 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=08a1406d2f624b54be9892ab8b1b7b3b] alter complete (version 3)
I20240502 21:27:09.426440 8403 consensus_queue.cc:1035] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:09.431385 8403 consensus_queue.cc:1035] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:09.492823 8339 consensus_queue.cc:1035] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:09.497455 8339 consensus_queue.cc:1035] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:09.502202 4291 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:09.502430 4291 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:09.505986 4291 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet a0781e3a014a483d91a523e7cf1434b7
I20240502 21:27:09.506783 7195 tablet_service.cc:1514] Processing DeleteTablet for tablet a0781e3a014a483d91a523e7cf1434b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:09 UTC) from {username='slave'} at 127.0.0.1:44652
I20240502 21:27:09.507150 4818 tablet_service.cc:1514] Processing DeleteTablet for tablet a0781e3a014a483d91a523e7cf1434b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:09 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:27:09.507453 8422 tablet_replica.cc:330] T a0781e3a014a483d91a523e7cf1434b7 P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:27:09.507645 7329 tablet_service.cc:1514] Processing DeleteTablet for tablet a0781e3a014a483d91a523e7cf1434b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:09 UTC) from {username='slave'} at 127.0.0.1:47078
I20240502 21:27:09.507802 8422 raft_consensus.cc:2229] T a0781e3a014a483d91a523e7cf1434b7 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:09.508019 8423 tablet_replica.cc:330] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:27:09.508051 8422 raft_consensus.cc:2258] T a0781e3a014a483d91a523e7cf1434b7 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:09.508251 8423 raft_consensus.cc:2229] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:09.508607 8423 raft_consensus.cc:2258] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:09.509620 8423 ts_tablet_manager.cc:1875] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:09.511862 8422 ts_tablet_manager.cc:1875] T a0781e3a014a483d91a523e7cf1434b7 P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:09.514747 8423 ts_tablet_manager.cc:1888] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:09.514942 8423 log.cc:1196] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/a0781e3a014a483d91a523e7cf1434b7
I20240502 21:27:09.515442 8423 ts_tablet_manager.cc:1909] T a0781e3a014a483d91a523e7cf1434b7 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:27:09.516997 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet a0781e3a014a483d91a523e7cf1434b7 (table rebalancer_dynamic_table_new_name [id=08a1406d2f624b54be9892ab8b1b7b3b]) successfully deleted
I20240502 21:27:09.520349 8424 tablet_replica.cc:330] T a0781e3a014a483d91a523e7cf1434b7 P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:27:09.520720 8424 raft_consensus.cc:2229] T a0781e3a014a483d91a523e7cf1434b7 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:09.521037 8424 raft_consensus.cc:2258] T a0781e3a014a483d91a523e7cf1434b7 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:09.522836 8422 ts_tablet_manager.cc:1888] T a0781e3a014a483d91a523e7cf1434b7 P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:09.522984 8422 log.cc:1196] T a0781e3a014a483d91a523e7cf1434b7 P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/a0781e3a014a483d91a523e7cf1434b7
I20240502 21:27:09.522966 8424 ts_tablet_manager.cc:1875] T a0781e3a014a483d91a523e7cf1434b7 P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:09.523479 8422 ts_tablet_manager.cc:1909] T a0781e3a014a483d91a523e7cf1434b7 P 39981a8dfe94462b9a73ba489af4c714: Deleting consensus metadata
I20240502 21:27:09.525149 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet a0781e3a014a483d91a523e7cf1434b7 (table rebalancer_dynamic_table_new_name [id=08a1406d2f624b54be9892ab8b1b7b3b]) successfully deleted
I20240502 21:27:09.528024 8424 ts_tablet_manager.cc:1888] T a0781e3a014a483d91a523e7cf1434b7 P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:09.528177 8424 log.cc:1196] T a0781e3a014a483d91a523e7cf1434b7 P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/a0781e3a014a483d91a523e7cf1434b7
I20240502 21:27:09.528571 8424 ts_tablet_manager.cc:1909] T a0781e3a014a483d91a523e7cf1434b7 P d0456c5e55a542128688904e003d403c: Deleting consensus metadata
I20240502 21:27:09.529521 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet a0781e3a014a483d91a523e7cf1434b7 (table rebalancer_dynamic_table_new_name [id=08a1406d2f624b54be9892ab8b1b7b3b]) successfully deleted
I20240502 21:27:09.608466 4291 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:09.615868 4422 tablet_service.cc:1467] Processing CreateTablet for tablet 388e147c25f349b095ea03a1958ce501 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d7af799efd664232b8d18fe9aea8b943]), partition=
I20240502 21:27:09.615960 7595 tablet_service.cc:1467] Processing CreateTablet for tablet 388e147c25f349b095ea03a1958ce501 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d7af799efd664232b8d18fe9aea8b943]), partition=
I20240502 21:27:09.616303 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 388e147c25f349b095ea03a1958ce501. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:09.616348 7595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 388e147c25f349b095ea03a1958ce501. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:09.616324 7195 tablet_service.cc:1467] Processing CreateTablet for tablet 388e147c25f349b095ea03a1958ce501 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d7af799efd664232b8d18fe9aea8b943]), partition=
I20240502 21:27:09.616686 7195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 388e147c25f349b095ea03a1958ce501. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:09.621104 8426 tablet_bootstrap.cc:492] T 388e147c25f349b095ea03a1958ce501 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:09.623040 8426 tablet_bootstrap.cc:654] T 388e147c25f349b095ea03a1958ce501 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:09.628466 8428 tablet_bootstrap.cc:492] T 388e147c25f349b095ea03a1958ce501 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:09.628970 8426 tablet_bootstrap.cc:492] T 388e147c25f349b095ea03a1958ce501 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:09.629179 8426 ts_tablet_manager.cc:1367] T 388e147c25f349b095ea03a1958ce501 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240502 21:27:09.631294 8426 raft_consensus.cc:348] T 388e147c25f349b095ea03a1958ce501 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:09.631531 8426 raft_consensus.cc:374] T 388e147c25f349b095ea03a1958ce501 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:09.631647 8426 raft_consensus.cc:729] T 388e147c25f349b095ea03a1958ce501 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:09.631903 8426 consensus_queue.cc:260] T 388e147c25f349b095ea03a1958ce501 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:09.632555 8426 ts_tablet_manager.cc:1398] T 388e147c25f349b095ea03a1958ce501 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:09.633829 8428 tablet_bootstrap.cc:654] T 388e147c25f349b095ea03a1958ce501 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:09.634231 8427 tablet_bootstrap.cc:492] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:09.636885 8427 tablet_bootstrap.cc:654] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:09.637360 8428 tablet_bootstrap.cc:492] T 388e147c25f349b095ea03a1958ce501 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:09.637531 8428 ts_tablet_manager.cc:1367] T 388e147c25f349b095ea03a1958ce501 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240502 21:27:09.638329 8428 raft_consensus.cc:348] T 388e147c25f349b095ea03a1958ce501 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:09.638610 8428 raft_consensus.cc:374] T 388e147c25f349b095ea03a1958ce501 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:09.638768 8428 raft_consensus.cc:729] T 388e147c25f349b095ea03a1958ce501 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:09.638901 8427 tablet_bootstrap.cc:492] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:09.639081 8427 ts_tablet_manager.cc:1367] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 21:27:09.639046 8428 consensus_queue.cc:260] T 388e147c25f349b095ea03a1958ce501 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:09.639916 8428 ts_tablet_manager.cc:1398] T 388e147c25f349b095ea03a1958ce501 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:09.640048 8427 raft_consensus.cc:348] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:09.640287 8427 raft_consensus.cc:374] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:09.640393 8427 raft_consensus.cc:729] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:09.640645 8427 consensus_queue.cc:260] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:09.641139 8427 ts_tablet_manager.cc:1398] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:09.670153 8168 consensus_queue.cc:1035] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:09.674793 8403 consensus_queue.cc:1035] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:09.788846 8434 raft_consensus.cc:482] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:09.789043 8434 raft_consensus.cc:504] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:09.790270 8434 leader_election.cc:288] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:09.791498 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "388e147c25f349b095ea03a1958ce501" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:27:09.791632 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "388e147c25f349b095ea03a1958ce501" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:09.792026 4440 raft_consensus.cc:2454] T 388e147c25f349b095ea03a1958ce501 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:09.792199 7215 raft_consensus.cc:2454] T 388e147c25f349b095ea03a1958ce501 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:09.793269 7549 leader_election.cc:302] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:09.794199 8434 raft_consensus.cc:2781] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:09.794466 8434 raft_consensus.cc:482] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:09.794653 8434 raft_consensus.cc:3037] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:09.798271 8434 raft_consensus.cc:504] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:09.798897 8434 leader_election.cc:288] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:09.799691 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "388e147c25f349b095ea03a1958ce501" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:27:09.799712 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "388e147c25f349b095ea03a1958ce501" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:09.799921 7215 raft_consensus.cc:3037] T 388e147c25f349b095ea03a1958ce501 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:09.799935 4440 raft_consensus.cc:3037] T 388e147c25f349b095ea03a1958ce501 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:09.801945 4440 raft_consensus.cc:2454] T 388e147c25f349b095ea03a1958ce501 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:09.802328 7549 leader_election.cc:302] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:09.802630 8434 raft_consensus.cc:2781] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:09.803092 8434 raft_consensus.cc:686] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Becoming Leader. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Running, Role: LEADER
I20240502 21:27:09.803352 8434 consensus_queue.cc:237] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:09.806617 4294 catalog_manager.cc:5484] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ddd486109f5434cace13d60d633ac04 (127.29.82.72). New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:09.807737 7215 raft_consensus.cc:2454] T 388e147c25f349b095ea03a1958ce501 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:09.873334 4294 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:09.882560 4422 tablet_service.cc:1467] Processing CreateTablet for tablet b589c7da65e34002b56cbf867d033c76 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=add5d1abdc46419a86ab49c59629bc2d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:09.882843 7462 tablet_service.cc:1467] Processing CreateTablet for tablet b589c7da65e34002b56cbf867d033c76 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=add5d1abdc46419a86ab49c59629bc2d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:09.883029 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b589c7da65e34002b56cbf867d033c76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:09.883306 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b589c7da65e34002b56cbf867d033c76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:09.888293 7195 tablet_service.cc:1467] Processing CreateTablet for tablet b589c7da65e34002b56cbf867d033c76 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=add5d1abdc46419a86ab49c59629bc2d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:09.888514 7329 tablet_service.cc:1467] Processing CreateTablet for tablet c6459a8c391249529d2672ab6c62a8af (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=add5d1abdc46419a86ab49c59629bc2d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:09.888763 7195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b589c7da65e34002b56cbf867d033c76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:09.888978 7329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6459a8c391249529d2672ab6c62a8af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:09.893455 8426 tablet_bootstrap.cc:492] T b589c7da65e34002b56cbf867d033c76 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:09.895236 8426 tablet_bootstrap.cc:654] T b589c7da65e34002b56cbf867d033c76 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:09.896562 8428 tablet_bootstrap.cc:492] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:09.897745 8428 tablet_bootstrap.cc:654] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:09.899883 4422 tablet_service.cc:1467] Processing CreateTablet for tablet c6459a8c391249529d2672ab6c62a8af (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=add5d1abdc46419a86ab49c59629bc2d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:09.900352 7328 tablet_service.cc:1467] Processing CreateTablet for tablet 616eda3cb218475c8bf0108cc6b90810 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=add5d1abdc46419a86ab49c59629bc2d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:09.900372 7595 tablet_service.cc:1467] Processing CreateTablet for tablet c6459a8c391249529d2672ab6c62a8af (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=add5d1abdc46419a86ab49c59629bc2d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:09.900496 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6459a8c391249529d2672ab6c62a8af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:09.900581 8426 tablet_bootstrap.cc:492] T b589c7da65e34002b56cbf867d033c76 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:09.900743 8426 ts_tablet_manager.cc:1367] T b589c7da65e34002b56cbf867d033c76 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.002s
I20240502 21:27:09.900786 7595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6459a8c391249529d2672ab6c62a8af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:09.900804 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 616eda3cb218475c8bf0108cc6b90810. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:09.901583 8426 raft_consensus.cc:348] T b589c7da65e34002b56cbf867d033c76 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:09.901830 8426 raft_consensus.cc:374] T b589c7da65e34002b56cbf867d033c76 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:09.901902 8426 raft_consensus.cc:729] T b589c7da65e34002b56cbf867d033c76 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:09.902143 8426 consensus_queue.cc:260] T b589c7da65e34002b56cbf867d033c76 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:09.902688 8426 ts_tablet_manager.cc:1398] T b589c7da65e34002b56cbf867d033c76 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:09.907490 7464 tablet_service.cc:1467] Processing CreateTablet for tablet 616eda3cb218475c8bf0108cc6b90810 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=add5d1abdc46419a86ab49c59629bc2d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:09.907619 7597 tablet_service.cc:1467] Processing CreateTablet for tablet 616eda3cb218475c8bf0108cc6b90810 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=add5d1abdc46419a86ab49c59629bc2d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:09.907964 7464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 616eda3cb218475c8bf0108cc6b90810. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:09.908036 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 616eda3cb218475c8bf0108cc6b90810. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:09.912537 8427 tablet_bootstrap.cc:492] T c6459a8c391249529d2672ab6c62a8af P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:09.914371 8427 tablet_bootstrap.cc:654] T c6459a8c391249529d2672ab6c62a8af P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:09.915247 8440 tablet_bootstrap.cc:492] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:09.917014 8427 tablet_bootstrap.cc:492] T c6459a8c391249529d2672ab6c62a8af P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:09.917188 8427 ts_tablet_manager.cc:1367] T c6459a8c391249529d2672ab6c62a8af P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:27:09.918206 8426 tablet_bootstrap.cc:492] T c6459a8c391249529d2672ab6c62a8af P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:09.918066 4298 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:09.918241 8427 raft_consensus.cc:348] T c6459a8c391249529d2672ab6c62a8af P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:09.918603 8427 raft_consensus.cc:374] T c6459a8c391249529d2672ab6c62a8af P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:09.918736 8427 raft_consensus.cc:729] T c6459a8c391249529d2672ab6c62a8af P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:09.919226 8440 tablet_bootstrap.cc:654] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:09.919062 8427 consensus_queue.cc:260] T c6459a8c391249529d2672ab6c62a8af P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:09.919904 8426 tablet_bootstrap.cc:654] T c6459a8c391249529d2672ab6c62a8af P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:09.921293 8428 tablet_bootstrap.cc:492] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:09.921528 8428 ts_tablet_manager.cc:1367] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20240502 21:27:09.922240 8427 ts_tablet_manager.cc:1398] T c6459a8c391249529d2672ab6c62a8af P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:27:09.922530 8427 tablet_bootstrap.cc:492] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:09.922538 8428 raft_consensus.cc:348] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:09.922834 8428 raft_consensus.cc:374] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:09.922917 8428 raft_consensus.cc:729] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:09.923485 8428 consensus_queue.cc:260] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:09.923887 8442 tablet_bootstrap.cc:492] T b589c7da65e34002b56cbf867d033c76 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:09.925549 8442 tablet_bootstrap.cc:654] T b589c7da65e34002b56cbf867d033c76 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:09.927708 8428 ts_tablet_manager.cc:1398] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 21:27:09.929556 8427 tablet_bootstrap.cc:654] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:09.938122 8442 tablet_bootstrap.cc:492] T b589c7da65e34002b56cbf867d033c76 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:09.938331 8442 ts_tablet_manager.cc:1367] T b589c7da65e34002b56cbf867d033c76 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.007s
I20240502 21:27:09.938717 8426 tablet_bootstrap.cc:492] T c6459a8c391249529d2672ab6c62a8af P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:09.938908 8426 ts_tablet_manager.cc:1367] T c6459a8c391249529d2672ab6c62a8af P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240502 21:27:09.939149 8442 raft_consensus.cc:348] T b589c7da65e34002b56cbf867d033c76 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:09.939052 8427 tablet_bootstrap.cc:492] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:09.939407 8442 raft_consensus.cc:374] T b589c7da65e34002b56cbf867d033c76 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:09.939548 8442 raft_consensus.cc:729] T b589c7da65e34002b56cbf867d033c76 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:09.939646 8427 ts_tablet_manager.cc:1367] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20240502 21:27:09.939844 8442 consensus_queue.cc:260] T b589c7da65e34002b56cbf867d033c76 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:09.939910 8426 raft_consensus.cc:348] T c6459a8c391249529d2672ab6c62a8af P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:09.940091 8426 raft_consensus.cc:374] T c6459a8c391249529d2672ab6c62a8af P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:09.940191 8426 raft_consensus.cc:729] T c6459a8c391249529d2672ab6c62a8af P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:09.940807 8426 consensus_queue.cc:260] T c6459a8c391249529d2672ab6c62a8af P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:09.941263 8442 ts_tablet_manager.cc:1398] T b589c7da65e34002b56cbf867d033c76 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:09.941480 8426 ts_tablet_manager.cc:1398] T c6459a8c391249529d2672ab6c62a8af P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:09.941589 8442 tablet_bootstrap.cc:492] T 616eda3cb218475c8bf0108cc6b90810 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:09.943136 8442 tablet_bootstrap.cc:654] T 616eda3cb218475c8bf0108cc6b90810 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:09.943267 7215 raft_consensus.cc:1261] T 388e147c25f349b095ea03a1958ce501 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:09.945046 8434 consensus_queue.cc:1035] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:09.947156 4440 raft_consensus.cc:1261] T 388e147c25f349b095ea03a1958ce501 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:09.947857 8434 consensus_queue.cc:1035] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:09.952096 8440 tablet_bootstrap.cc:492] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:09.952278 8440 ts_tablet_manager.cc:1367] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.006s
I20240502 21:27:09.953181 8440 raft_consensus.cc:348] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:09.953529 8440 raft_consensus.cc:374] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:09.953696 8440 raft_consensus.cc:729] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:09.953662 4294 catalog_manager.cc:5576] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193) does not have the latest schema for tablet 388e147c25f349b095ea03a1958ce501 (table rebalancer_dynamic_table [id=d7af799efd664232b8d18fe9aea8b943]). Expected version 1 got 0
I20240502 21:27:09.954053 8440 consensus_queue.cc:260] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:09.957610 8440 ts_tablet_manager.cc:1398] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:09.958026 8440 tablet_bootstrap.cc:492] T 616eda3cb218475c8bf0108cc6b90810 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:09.960544 8440 tablet_bootstrap.cc:654] T 616eda3cb218475c8bf0108cc6b90810 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:09.962637 8451 tablet.cc:1722] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:09.963297 8452 tablet.cc:1722] T 388e147c25f349b095ea03a1958ce501 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:09.964303 8451 tablet.cc:1620] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04: MemRowSet was empty: no flush needed.
I20240502 21:27:09.964913 8452 tablet.cc:1620] T 388e147c25f349b095ea03a1958ce501 P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:09.964841 8427 raft_consensus.cc:348] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:09.965133 8427 raft_consensus.cc:374] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:09.965510 8427 raft_consensus.cc:729] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:09.966012 8427 consensus_queue.cc:260] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:09.966554 4298 catalog_manager.cc:5576] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193) does not have the latest schema for tablet 388e147c25f349b095ea03a1958ce501 (table rebalancer_dynamic_table [id=d7af799efd664232b8d18fe9aea8b943]). Expected version 1 got 0
I20240502 21:27:09.967250 8427 ts_tablet_manager.cc:1398] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20240502 21:27:09.983016 8440 tablet_bootstrap.cc:492] T 616eda3cb218475c8bf0108cc6b90810 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:09.983017 4291 catalog_manager.cc:5576] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781) does not have the latest schema for tablet 388e147c25f349b095ea03a1958ce501 (table rebalancer_dynamic_table [id=d7af799efd664232b8d18fe9aea8b943]). Expected version 1 got 0
I20240502 21:27:09.983215 8440 ts_tablet_manager.cc:1367] T 616eda3cb218475c8bf0108cc6b90810 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.003s
I20240502 21:27:09.984158 8440 raft_consensus.cc:348] T 616eda3cb218475c8bf0108cc6b90810 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:09.984431 8440 raft_consensus.cc:374] T 616eda3cb218475c8bf0108cc6b90810 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:09.984527 8440 raft_consensus.cc:729] T 616eda3cb218475c8bf0108cc6b90810 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:09.984808 8440 consensus_queue.cc:260] T 616eda3cb218475c8bf0108cc6b90810 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:09.985574 8440 ts_tablet_manager.cc:1398] T 616eda3cb218475c8bf0108cc6b90810 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:09.987648 8292 raft_consensus.cc:482] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:09.987792 8292 raft_consensus.cc:504] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:09.988605 8292 leader_election.cc:288] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:09.989348 4298 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=d7af799efd664232b8d18fe9aea8b943] alter complete (version 1)
I20240502 21:27:09.991236 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6459a8c391249529d2672ab6c62a8af" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:27:09.991514 4442 raft_consensus.cc:2454] T c6459a8c391249529d2672ab6c62a8af P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 0.
I20240502 21:27:09.992378 7281 leader_election.cc:302] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:09.992903 8292 raft_consensus.cc:2781] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:09.993052 8292 raft_consensus.cc:482] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:09.992985 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6459a8c391249529d2672ab6c62a8af" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:09.993155 8292 raft_consensus.cc:3037] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:09.993213 7618 raft_consensus.cc:2454] T c6459a8c391249529d2672ab6c62a8af P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 0.
I20240502 21:27:09.996482 8292 raft_consensus.cc:504] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:09.998668 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6459a8c391249529d2672ab6c62a8af" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:27:09.998885 4442 raft_consensus.cc:3037] T c6459a8c391249529d2672ab6c62a8af P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.001147 4442 raft_consensus.cc:2454] T c6459a8c391249529d2672ab6c62a8af P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
I20240502 21:27:10.001721 7281 leader_election.cc:302] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:10.001736 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6459a8c391249529d2672ab6c62a8af" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:10.001952 7618 raft_consensus.cc:3037] T c6459a8c391249529d2672ab6c62a8af P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.002072 8325 raft_consensus.cc:2781] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:10.004959 8292 leader_election.cc:288] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:10.005470 7618 raft_consensus.cc:2454] T c6459a8c391249529d2672ab6c62a8af P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
I20240502 21:27:10.007886 8325 raft_consensus.cc:686] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Becoming Leader. State: Replica: d0456c5e55a542128688904e003d403c, State: Running, Role: LEADER
I20240502 21:27:10.008157 8325 consensus_queue.cc:237] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:10.011515 4298 catalog_manager.cc:5484] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c reported cstate change: term changed from 0 to 1, leader changed from <none> to d0456c5e55a542128688904e003d403c (127.29.82.70). New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:10.017330 8454 tablet.cc:1722] T 388e147c25f349b095ea03a1958ce501 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:10.017966 8454 tablet.cc:1620] T 388e147c25f349b095ea03a1958ce501 P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:10.021706 4298 catalog_manager.cc:5576] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781) does not have the latest schema for tablet 388e147c25f349b095ea03a1958ce501 (table rebalancer_dynamic_table [id=d7af799efd664232b8d18fe9aea8b943]). Expected version 1 got 0
I20240502 21:27:10.022352 8339 raft_consensus.cc:482] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:10.022584 8339 raft_consensus.cc:504] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:10.023511 8339 leader_election.cc:288] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:10.024653 8168 raft_consensus.cc:482] T b589c7da65e34002b56cbf867d033c76 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:10.024873 8168 raft_consensus.cc:504] T b589c7da65e34002b56cbf867d033c76 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:10.025707 8168 leader_election.cc:288] T b589c7da65e34002b56cbf867d033c76 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:10.026444 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b589c7da65e34002b56cbf867d033c76" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:10.026681 7215 raft_consensus.cc:2454] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:10.027451 4377 leader_election.cc:302] T b589c7da65e34002b56cbf867d033c76 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:10.028283 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b589c7da65e34002b56cbf867d033c76" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:10.028537 7484 raft_consensus.cc:2454] T b589c7da65e34002b56cbf867d033c76 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:10.029033 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b589c7da65e34002b56cbf867d033c76" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:10.029224 7483 raft_consensus.cc:2454] T b589c7da65e34002b56cbf867d033c76 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:10.029372 8442 tablet_bootstrap.cc:492] T 616eda3cb218475c8bf0108cc6b90810 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:10.029635 8442 ts_tablet_manager.cc:1367] T 616eda3cb218475c8bf0108cc6b90810 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.088s user 0.031s sys 0.049s
I20240502 21:27:10.029752 7148 leader_election.cc:302] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:10.030284 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b589c7da65e34002b56cbf867d033c76" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:27:10.030490 4443 raft_consensus.cc:2454] T b589c7da65e34002b56cbf867d033c76 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:10.030529 8442 raft_consensus.cc:348] T 616eda3cb218475c8bf0108cc6b90810 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:10.030768 8442 raft_consensus.cc:374] T 616eda3cb218475c8bf0108cc6b90810 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.030855 8442 raft_consensus.cc:729] T 616eda3cb218475c8bf0108cc6b90810 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.031124 8442 consensus_queue.cc:260] T 616eda3cb218475c8bf0108cc6b90810 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:10.032194 8403 raft_consensus.cc:2781] T b589c7da65e34002b56cbf867d033c76 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:10.032301 8403 raft_consensus.cc:482] T b589c7da65e34002b56cbf867d033c76 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:10.032407 8403 raft_consensus.cc:3037] T b589c7da65e34002b56cbf867d033c76 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.032718 8442 ts_tablet_manager.cc:1398] T 616eda3cb218475c8bf0108cc6b90810 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:27:10.035017 8403 raft_consensus.cc:504] T b589c7da65e34002b56cbf867d033c76 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:10.035702 8419 raft_consensus.cc:2781] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:10.035825 8419 raft_consensus.cc:482] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:10.035909 8419 raft_consensus.cc:3037] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.038513 8419 raft_consensus.cc:504] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:10.039242 8419 leader_election.cc:288] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:10.041882 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b589c7da65e34002b56cbf867d033c76" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:10.042218 7215 raft_consensus.cc:2379] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 71e1375b799b439fa3c83ca763c039c1 in current term 1: Already voted for candidate 39981a8dfe94462b9a73ba489af4c714 in this term.
I20240502 21:27:10.042759 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b589c7da65e34002b56cbf867d033c76" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:10.043011 7484 raft_consensus.cc:3037] T b589c7da65e34002b56cbf867d033c76 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.046751 7484 raft_consensus.cc:2454] T b589c7da65e34002b56cbf867d033c76 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:10.048003 8403 leader_election.cc:288] T b589c7da65e34002b56cbf867d033c76 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:10.048897 4375 leader_election.cc:302] T b589c7da65e34002b56cbf867d033c76 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 71e1375b799b439fa3c83ca763c039c1; no voters: 39981a8dfe94462b9a73ba489af4c714
I20240502 21:27:10.049451 8403 raft_consensus.cc:2781] T b589c7da65e34002b56cbf867d033c76 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:10.049809 8403 raft_consensus.cc:686] T b589c7da65e34002b56cbf867d033c76 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:27:10.050719 8434 raft_consensus.cc:482] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:10.050688 8403 consensus_queue.cc:237] T b589c7da65e34002b56cbf867d033c76 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:10.050952 8434 raft_consensus.cc:504] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:10.052224 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616eda3cb218475c8bf0108cc6b90810" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:10.052484 7349 raft_consensus.cc:2454] T 616eda3cb218475c8bf0108cc6b90810 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:10.053337 7551 leader_election.cc:302] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:10.053661 7942 raft_consensus.cc:2781] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:10.053752 7942 raft_consensus.cc:482] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:10.053874 7942 raft_consensus.cc:3037] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.058256 7942 raft_consensus.cc:504] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:10.061444 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616eda3cb218475c8bf0108cc6b90810" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:10.061678 7349 raft_consensus.cc:3037] T 616eda3cb218475c8bf0108cc6b90810 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.061936 8434 leader_election.cc:288] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:10.061965 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b589c7da65e34002b56cbf867d033c76" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:10.062224 7483 raft_consensus.cc:2379] T b589c7da65e34002b56cbf867d033c76 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 39981a8dfe94462b9a73ba489af4c714 in current term 1: Already voted for candidate 71e1375b799b439fa3c83ca763c039c1 in this term.
I20240502 21:27:10.062218 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616eda3cb218475c8bf0108cc6b90810" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:10.062446 7484 raft_consensus.cc:3037] T 616eda3cb218475c8bf0108cc6b90810 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.062419 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b589c7da65e34002b56cbf867d033c76" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:27:10.063161 7148 leader_election.cc:302] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714; no voters: 31a72286397c4edc9a63eca3412be168, 71e1375b799b439fa3c83ca763c039c1
I20240502 21:27:10.063481 8457 raft_consensus.cc:2726] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 21:27:10.063762 7942 leader_election.cc:288] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:10.064407 7349 raft_consensus.cc:2454] T 616eda3cb218475c8bf0108cc6b90810 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:10.065009 7551 leader_election.cc:302] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:10.065125 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616eda3cb218475c8bf0108cc6b90810" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:10.065356 7942 raft_consensus.cc:2781] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:10.065557 7942 raft_consensus.cc:686] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Becoming Leader. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Running, Role: LEADER
I20240502 21:27:10.066192 7942 consensus_queue.cc:237] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:10.066368 7484 raft_consensus.cc:2454] T 616eda3cb218475c8bf0108cc6b90810 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:10.068048 4294 catalog_manager.cc:5484] T b589c7da65e34002b56cbf867d033c76 P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:10.071964 4298 catalog_manager.cc:5484] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ddd486109f5434cace13d60d633ac04 (127.29.82.72). New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:10.101363 4298 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:10.108532 8451 tablet.cc:1722] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:10.109093 8451 tablet.cc:1620] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04: MemRowSet was empty: no flush needed.
I20240502 21:27:10.109059 8454 tablet.cc:1722] T 388e147c25f349b095ea03a1958ce501 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:10.109536 8454 tablet.cc:1620] T 388e147c25f349b095ea03a1958ce501 P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:10.116199 8452 tablet.cc:1722] T 388e147c25f349b095ea03a1958ce501 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:10.116693 8452 tablet.cc:1620] T 388e147c25f349b095ea03a1958ce501 P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:10.120510 4298 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=d7af799efd664232b8d18fe9aea8b943] alter complete (version 2)
I20240502 21:27:10.223284 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:10.231119 8451 tablet.cc:1722] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:10.232012 8452 tablet.cc:1722] T 388e147c25f349b095ea03a1958ce501 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:10.232287 8454 tablet.cc:1722] T 388e147c25f349b095ea03a1958ce501 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:10.239682 4294 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=d7af799efd664232b8d18fe9aea8b943] alter complete (version 3)
I20240502 21:27:10.342178 4298 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:10.342434 4298 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:10.345844 4298 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 388e147c25f349b095ea03a1958ce501
I20240502 21:27:10.346626 4422 tablet_service.cc:1514] Processing DeleteTablet for tablet 388e147c25f349b095ea03a1958ce501 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:10 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:10.346768 7595 tablet_service.cc:1514] Processing DeleteTablet for tablet 388e147c25f349b095ea03a1958ce501 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:10 UTC) from {username='slave'} at 127.0.0.1:44444
I20240502 21:27:10.347085 7195 tablet_service.cc:1514] Processing DeleteTablet for tablet 388e147c25f349b095ea03a1958ce501 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:10 UTC) from {username='slave'} at 127.0.0.1:44652
I20240502 21:27:10.358937 8465 tablet_replica.cc:330] T 388e147c25f349b095ea03a1958ce501 P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:27:10.359165 8465 raft_consensus.cc:2229] T 388e147c25f349b095ea03a1958ce501 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:10.359402 8465 raft_consensus.cc:2258] T 388e147c25f349b095ea03a1958ce501 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:10.359786 8463 tablet_replica.cc:330] T 388e147c25f349b095ea03a1958ce501 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:10.360211 8463 raft_consensus.cc:2229] T 388e147c25f349b095ea03a1958ce501 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:10.360426 8465 ts_tablet_manager.cc:1875] T 388e147c25f349b095ea03a1958ce501 P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:10.360541 8463 raft_consensus.cc:2258] T 388e147c25f349b095ea03a1958ce501 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:10.361809 8463 ts_tablet_manager.cc:1875] T 388e147c25f349b095ea03a1958ce501 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:10.363284 8464 tablet_replica.cc:330] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04: stopping tablet replica
I20240502 21:27:10.363619 8464 raft_consensus.cc:2229] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:10.364140 8464 raft_consensus.cc:2258] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:10.365257 8464 ts_tablet_manager.cc:1875] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:10.366719 8465 ts_tablet_manager.cc:1888] T 388e147c25f349b095ea03a1958ce501 P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:10.366883 8465 log.cc:1196] T 388e147c25f349b095ea03a1958ce501 P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/388e147c25f349b095ea03a1958ce501
I20240502 21:27:10.367376 8465 ts_tablet_manager.cc:1909] T 388e147c25f349b095ea03a1958ce501 P 39981a8dfe94462b9a73ba489af4c714: Deleting consensus metadata
I20240502 21:27:10.368561 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet 388e147c25f349b095ea03a1958ce501 (table rebalancer_dynamic_table_new_name [id=d7af799efd664232b8d18fe9aea8b943]) successfully deleted
I20240502 21:27:10.368952 8463 ts_tablet_manager.cc:1888] T 388e147c25f349b095ea03a1958ce501 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:10.369117 8463 log.cc:1196] T 388e147c25f349b095ea03a1958ce501 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/388e147c25f349b095ea03a1958ce501
I20240502 21:27:10.369696 8463 ts_tablet_manager.cc:1909] T 388e147c25f349b095ea03a1958ce501 P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:10.371330 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet 388e147c25f349b095ea03a1958ce501 (table rebalancer_dynamic_table_new_name [id=d7af799efd664232b8d18fe9aea8b943]) successfully deleted
I20240502 21:27:10.374178 8464 ts_tablet_manager.cc:1888] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:10.374286 8464 log.cc:1196] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/wals/388e147c25f349b095ea03a1958ce501
I20240502 21:27:10.374702 8464 ts_tablet_manager.cc:1909] T 388e147c25f349b095ea03a1958ce501 P 0ddd486109f5434cace13d60d633ac04: Deleting consensus metadata
I20240502 21:27:10.375650 4285 catalog_manager.cc:4830] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): tablet 388e147c25f349b095ea03a1958ce501 (table rebalancer_dynamic_table_new_name [id=d7af799efd664232b8d18fe9aea8b943]) successfully deleted
I20240502 21:27:10.451849 4298 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:10.459231 7195 tablet_service.cc:1467] Processing CreateTablet for tablet 55466814dd2a4396a37aa37e93d9119f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2cb60813e32144149690f4e87b2ad229]), partition=
I20240502 21:27:10.459249 7462 tablet_service.cc:1467] Processing CreateTablet for tablet 55466814dd2a4396a37aa37e93d9119f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2cb60813e32144149690f4e87b2ad229]), partition=
I20240502 21:27:10.459635 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55466814dd2a4396a37aa37e93d9119f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.459595 7329 tablet_service.cc:1467] Processing CreateTablet for tablet 55466814dd2a4396a37aa37e93d9119f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2cb60813e32144149690f4e87b2ad229]), partition=
I20240502 21:27:10.459698 7195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55466814dd2a4396a37aa37e93d9119f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.459980 7329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55466814dd2a4396a37aa37e93d9119f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.464040 8442 tablet_bootstrap.cc:492] T 55466814dd2a4396a37aa37e93d9119f P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:10.465550 8442 tablet_bootstrap.cc:654] T 55466814dd2a4396a37aa37e93d9119f P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.466159 8440 tablet_bootstrap.cc:492] T 55466814dd2a4396a37aa37e93d9119f P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:10.468128 8440 tablet_bootstrap.cc:654] T 55466814dd2a4396a37aa37e93d9119f P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.472867 8440 tablet_bootstrap.cc:492] T 55466814dd2a4396a37aa37e93d9119f P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:10.473100 8440 ts_tablet_manager.cc:1367] T 55466814dd2a4396a37aa37e93d9119f P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240502 21:27:10.473928 8440 raft_consensus.cc:348] T 55466814dd2a4396a37aa37e93d9119f P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:10.474206 8440 raft_consensus.cc:374] T 55466814dd2a4396a37aa37e93d9119f P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.474315 8440 raft_consensus.cc:729] T 55466814dd2a4396a37aa37e93d9119f P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.474606 8440 consensus_queue.cc:260] T 55466814dd2a4396a37aa37e93d9119f P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:10.476096 8440 ts_tablet_manager.cc:1398] T 55466814dd2a4396a37aa37e93d9119f P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:27:10.476375 8467 tablet_bootstrap.cc:492] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:10.477746 8467 tablet_bootstrap.cc:654] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.479835 8467 tablet_bootstrap.cc:492] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:10.480023 8467 ts_tablet_manager.cc:1367] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:27:10.480839 8467 raft_consensus.cc:348] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:10.481097 8467 raft_consensus.cc:374] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.481238 8467 raft_consensus.cc:729] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.481508 8467 consensus_queue.cc:260] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:10.481621 8442 tablet_bootstrap.cc:492] T 55466814dd2a4396a37aa37e93d9119f P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:10.481782 8442 ts_tablet_manager.cc:1367] T 55466814dd2a4396a37aa37e93d9119f P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.011s
I20240502 21:27:10.482092 8467 ts_tablet_manager.cc:1398] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:10.482559 8442 raft_consensus.cc:348] T 55466814dd2a4396a37aa37e93d9119f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:10.482779 8442 raft_consensus.cc:374] T 55466814dd2a4396a37aa37e93d9119f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.482877 8442 raft_consensus.cc:729] T 55466814dd2a4396a37aa37e93d9119f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.483106 8442 consensus_queue.cc:260] T 55466814dd2a4396a37aa37e93d9119f P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:10.483661 8442 ts_tablet_manager.cc:1398] T 55466814dd2a4396a37aa37e93d9119f P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:10.510135 8325 consensus_queue.cc:1035] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:10.515067 8325 consensus_queue.cc:1035] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:10.537315 8434 consensus_queue.cc:1035] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:10.541312 8434 consensus_queue.cc:1035] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:10.592221 4291 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:10.600710 7462 tablet_service.cc:1467] Processing CreateTablet for tablet 80304ab48274408393c2e245477c3a78 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=1e6edd8bfe59407fa7863cd780bfae31]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:10.601140 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80304ab48274408393c2e245477c3a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.601948 7329 tablet_service.cc:1467] Processing CreateTablet for tablet 80304ab48274408393c2e245477c3a78 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=1e6edd8bfe59407fa7863cd780bfae31]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:10.602361 7329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80304ab48274408393c2e245477c3a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.602349 7463 tablet_service.cc:1467] Processing CreateTablet for tablet a85e7872c28a410496f31ecff9c71edb (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=1e6edd8bfe59407fa7863cd780bfae31]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:10.602735 7463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a85e7872c28a410496f31ecff9c71edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.603974 7464 tablet_service.cc:1467] Processing CreateTablet for tablet 796c7f3cc1f9426daecc89add3c6aa57 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=1e6edd8bfe59407fa7863cd780bfae31]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:10.604363 7464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 796c7f3cc1f9426daecc89add3c6aa57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.600430 7595 tablet_service.cc:1467] Processing CreateTablet for tablet 80304ab48274408393c2e245477c3a78 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=1e6edd8bfe59407fa7863cd780bfae31]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:10.606304 7595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80304ab48274408393c2e245477c3a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.609553 8442 tablet_bootstrap.cc:492] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:10.609745 4422 tablet_service.cc:1467] Processing CreateTablet for tablet a85e7872c28a410496f31ecff9c71edb (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=1e6edd8bfe59407fa7863cd780bfae31]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:10.610190 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a85e7872c28a410496f31ecff9c71edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.611275 8442 tablet_bootstrap.cc:654] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.611295 4555 tablet_service.cc:1467] Processing CreateTablet for tablet 796c7f3cc1f9426daecc89add3c6aa57 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=1e6edd8bfe59407fa7863cd780bfae31]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:10.611778 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 796c7f3cc1f9426daecc89add3c6aa57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.616062 7596 tablet_service.cc:1467] Processing CreateTablet for tablet a85e7872c28a410496f31ecff9c71edb (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=1e6edd8bfe59407fa7863cd780bfae31]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:10.616609 7596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a85e7872c28a410496f31ecff9c71edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.619942 7597 tablet_service.cc:1467] Processing CreateTablet for tablet 796c7f3cc1f9426daecc89add3c6aa57 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=1e6edd8bfe59407fa7863cd780bfae31]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:10.620409 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 796c7f3cc1f9426daecc89add3c6aa57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.623530 8403 consensus_queue.cc:1035] T b589c7da65e34002b56cbf867d033c76 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:10.625957 8479 tablet_bootstrap.cc:492] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:10.629645 8479 tablet_bootstrap.cc:654] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.631561 8482 tablet_bootstrap.cc:492] T 80304ab48274408393c2e245477c3a78 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:10.632900 8440 tablet_bootstrap.cc:492] T 80304ab48274408393c2e245477c3a78 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:10.634883 8440 tablet_bootstrap.cc:654] T 80304ab48274408393c2e245477c3a78 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.636904 8440 tablet_bootstrap.cc:492] T 80304ab48274408393c2e245477c3a78 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:10.637077 8440 ts_tablet_manager.cc:1367] T 80304ab48274408393c2e245477c3a78 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240502 21:27:10.637362 8480 tablet_bootstrap.cc:492] T a85e7872c28a410496f31ecff9c71edb P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:10.638087 8442 tablet_bootstrap.cc:492] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:10.638269 8442 ts_tablet_manager.cc:1367] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.014s
I20240502 21:27:10.639480 8480 tablet_bootstrap.cc:654] T a85e7872c28a410496f31ecff9c71edb P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.639969 8442 raft_consensus.cc:348] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:10.640208 8442 raft_consensus.cc:374] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.640296 8442 raft_consensus.cc:729] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.640996 8442 consensus_queue.cc:260] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:10.638278 8440 raft_consensus.cc:348] T 80304ab48274408393c2e245477c3a78 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:10.641422 8440 raft_consensus.cc:374] T 80304ab48274408393c2e245477c3a78 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.641536 8440 raft_consensus.cc:729] T 80304ab48274408393c2e245477c3a78 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.642570 8440 consensus_queue.cc:260] T 80304ab48274408393c2e245477c3a78 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:10.647485 8442 ts_tablet_manager.cc:1398] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240502 21:27:10.647972 8440 ts_tablet_manager.cc:1398] T 80304ab48274408393c2e245477c3a78 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.011s user 0.002s sys 0.001s
I20240502 21:27:10.650336 8325 raft_consensus.cc:482] T 80304ab48274408393c2e245477c3a78 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:10.650486 8325 raft_consensus.cc:504] T 80304ab48274408393c2e245477c3a78 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:10.651167 8325 leader_election.cc:288] T 80304ab48274408393c2e245477c3a78 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:10.651875 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80304ab48274408393c2e245477c3a78" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
W20240502 21:27:10.652652 7281 leader_election.cc:341] T 80304ab48274408393c2e245477c3a78 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:10.652798 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80304ab48274408393c2e245477c3a78" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
W20240502 21:27:10.653576 7282 leader_election.cc:341] T 80304ab48274408393c2e245477c3a78 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:10.653714 7282 leader_election.cc:302] T 80304ab48274408393c2e245477c3a78 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0456c5e55a542128688904e003d403c; no voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168
I20240502 21:27:10.654158 8325 raft_consensus.cc:2726] T 80304ab48274408393c2e245477c3a78 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:27:10.654577 8442 tablet_bootstrap.cc:492] T 796c7f3cc1f9426daecc89add3c6aa57 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:10.656338 8442 tablet_bootstrap.cc:654] T 796c7f3cc1f9426daecc89add3c6aa57 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.659909 8479 tablet_bootstrap.cc:492] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:10.661052 8480 tablet_bootstrap.cc:492] T a85e7872c28a410496f31ecff9c71edb P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:10.661290 8479 ts_tablet_manager.cc:1367] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.000s
I20240502 21:27:10.661300 8480 ts_tablet_manager.cc:1367] T a85e7872c28a410496f31ecff9c71edb P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.007s
I20240502 21:27:10.662544 8479 raft_consensus.cc:348] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:10.664676 8442 tablet_bootstrap.cc:492] T 796c7f3cc1f9426daecc89add3c6aa57 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:10.664710 8479 raft_consensus.cc:374] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.664809 8442 ts_tablet_manager.cc:1367] T 796c7f3cc1f9426daecc89add3c6aa57 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20240502 21:27:10.664827 8479 raft_consensus.cc:729] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.665146 8482 tablet_bootstrap.cc:654] T 80304ab48274408393c2e245477c3a78 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.665077 8479 consensus_queue.cc:260] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:10.665465 8442 raft_consensus.cc:348] T 796c7f3cc1f9426daecc89add3c6aa57 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:10.665617 8442 raft_consensus.cc:374] T 796c7f3cc1f9426daecc89add3c6aa57 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.665706 8442 raft_consensus.cc:729] T 796c7f3cc1f9426daecc89add3c6aa57 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.665905 8442 consensus_queue.cc:260] T 796c7f3cc1f9426daecc89add3c6aa57 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:10.666438 8442 ts_tablet_manager.cc:1398] T 796c7f3cc1f9426daecc89add3c6aa57 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:10.666823 8442 tablet_bootstrap.cc:492] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:10.666728 8488 consensus_queue.cc:1035] T b589c7da65e34002b56cbf867d033c76 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:10.668154 8479 ts_tablet_manager.cc:1398] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 21:27:10.668260 8480 raft_consensus.cc:348] T a85e7872c28a410496f31ecff9c71edb P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:10.668494 8442 tablet_bootstrap.cc:654] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.668545 8480 raft_consensus.cc:374] T a85e7872c28a410496f31ecff9c71edb P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.668655 8480 raft_consensus.cc:729] T a85e7872c28a410496f31ecff9c71edb P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.668931 8480 consensus_queue.cc:260] T a85e7872c28a410496f31ecff9c71edb P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:10.669685 8480 ts_tablet_manager.cc:1398] T a85e7872c28a410496f31ecff9c71edb P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 21:27:10.693307 8442 tablet_bootstrap.cc:492] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:10.693527 8442 ts_tablet_manager.cc:1367] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.010s
I20240502 21:27:10.693892 8482 tablet_bootstrap.cc:492] T 80304ab48274408393c2e245477c3a78 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:10.694056 8482 ts_tablet_manager.cc:1367] T 80304ab48274408393c2e245477c3a78 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.063s user 0.008s sys 0.009s
I20240502 21:27:10.694444 8442 raft_consensus.cc:348] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:10.694703 8442 raft_consensus.cc:374] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.694795 8442 raft_consensus.cc:729] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.694885 8482 raft_consensus.cc:348] T 80304ab48274408393c2e245477c3a78 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:10.695107 8482 raft_consensus.cc:374] T 80304ab48274408393c2e245477c3a78 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.695211 8482 raft_consensus.cc:729] T 80304ab48274408393c2e245477c3a78 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.695086 8442 consensus_queue.cc:260] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:10.695755 8442 ts_tablet_manager.cc:1398] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 21:27:10.695931 8482 consensus_queue.cc:260] T 80304ab48274408393c2e245477c3a78 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:10.697326 8482 ts_tablet_manager.cc:1398] T 80304ab48274408393c2e245477c3a78 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:27:10.697647 8482 tablet_bootstrap.cc:492] T 796c7f3cc1f9426daecc89add3c6aa57 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:10.700493 8482 tablet_bootstrap.cc:654] T 796c7f3cc1f9426daecc89add3c6aa57 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.702490 8482 tablet_bootstrap.cc:492] T 796c7f3cc1f9426daecc89add3c6aa57 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:10.702730 8482 ts_tablet_manager.cc:1367] T 796c7f3cc1f9426daecc89add3c6aa57 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240502 21:27:10.703662 8482 raft_consensus.cc:348] T 796c7f3cc1f9426daecc89add3c6aa57 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:10.703845 8482 raft_consensus.cc:374] T 796c7f3cc1f9426daecc89add3c6aa57 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.703928 8482 raft_consensus.cc:729] T 796c7f3cc1f9426daecc89add3c6aa57 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.704183 8482 consensus_queue.cc:260] T 796c7f3cc1f9426daecc89add3c6aa57 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:10.704787 8482 ts_tablet_manager.cc:1398] T 796c7f3cc1f9426daecc89add3c6aa57 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:10.705382 8482 tablet_bootstrap.cc:492] T a85e7872c28a410496f31ecff9c71edb P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:10.706614 8482 tablet_bootstrap.cc:654] T a85e7872c28a410496f31ecff9c71edb P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.710755 8482 tablet_bootstrap.cc:492] T a85e7872c28a410496f31ecff9c71edb P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:10.710893 8482 ts_tablet_manager.cc:1367] T a85e7872c28a410496f31ecff9c71edb P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240502 21:27:10.711468 8482 raft_consensus.cc:348] T a85e7872c28a410496f31ecff9c71edb P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:10.711725 8482 raft_consensus.cc:374] T a85e7872c28a410496f31ecff9c71edb P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.711839 8482 raft_consensus.cc:729] T a85e7872c28a410496f31ecff9c71edb P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.712067 8482 consensus_queue.cc:260] T a85e7872c28a410496f31ecff9c71edb P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:10.712648 8482 ts_tablet_manager.cc:1398] T a85e7872c28a410496f31ecff9c71edb P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:10.729682 8457 raft_consensus.cc:482] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:10.729863 8457 raft_consensus.cc:504] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:10.730528 8457 leader_election.cc:288] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:10.731123 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55466814dd2a4396a37aa37e93d9119f" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:10.731246 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55466814dd2a4396a37aa37e93d9119f" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:10.731405 7484 raft_consensus.cc:2454] T 55466814dd2a4396a37aa37e93d9119f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:10.731503 7349 raft_consensus.cc:2454] T 55466814dd2a4396a37aa37e93d9119f P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:10.731979 7150 leader_election.cc:302] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:10.732306 8457 raft_consensus.cc:2781] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:10.732414 8457 raft_consensus.cc:482] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:10.732527 8457 raft_consensus.cc:3037] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.734972 8457 raft_consensus.cc:504] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:10.736042 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55466814dd2a4396a37aa37e93d9119f" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:10.736143 8457 leader_election.cc:288] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:10.736141 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55466814dd2a4396a37aa37e93d9119f" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:10.736266 7484 raft_consensus.cc:3037] T 55466814dd2a4396a37aa37e93d9119f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.736353 7349 raft_consensus.cc:3037] T 55466814dd2a4396a37aa37e93d9119f P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.738516 7349 raft_consensus.cc:2454] T 55466814dd2a4396a37aa37e93d9119f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:10.738921 7150 leader_election.cc:302] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:10.739941 7484 raft_consensus.cc:2454] T 55466814dd2a4396a37aa37e93d9119f P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:10.740299 8457 raft_consensus.cc:2781] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:10.740816 8457 raft_consensus.cc:686] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:27:10.741079 8457 consensus_queue.cc:237] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:10.744186 4300 catalog_manager.cc:5484] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69). New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:10.765559 8180 raft_consensus.cc:482] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:10.765744 8180 raft_consensus.cc:504] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:10.766614 8180 leader_election.cc:288] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:10.766976 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "796c7f3cc1f9426daecc89add3c6aa57" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:10.767206 7484 raft_consensus.cc:2454] T 796c7f3cc1f9426daecc89add3c6aa57 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:27:10.767614 4508 leader_election.cc:302] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:27:10.767969 8180 raft_consensus.cc:2781] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:10.768111 8180 raft_consensus.cc:482] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:10.768232 8180 raft_consensus.cc:3037] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.770264 8180 raft_consensus.cc:504] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:10.770828 8180 leader_election.cc:288] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Requested vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:10.771252 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "796c7f3cc1f9426daecc89add3c6aa57" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:10.771457 7484 raft_consensus.cc:3037] T 796c7f3cc1f9426daecc89add3c6aa57 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.774941 7484 raft_consensus.cc:2454] T 796c7f3cc1f9426daecc89add3c6aa57 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:27:10.775408 4508 leader_election.cc:302] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:27:10.775717 8425 raft_consensus.cc:2781] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:10.775875 8425 raft_consensus.cc:686] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:27:10.776250 8425 consensus_queue.cc:237] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:10.779470 4291 catalog_manager.cc:5484] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:10.794518 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "796c7f3cc1f9426daecc89add3c6aa57" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:10.794795 7618 raft_consensus.cc:2454] T 796c7f3cc1f9426daecc89add3c6aa57 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:27:10.795115 7617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "796c7f3cc1f9426daecc89add3c6aa57" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:10.795326 7617 raft_consensus.cc:3037] T 796c7f3cc1f9426daecc89add3c6aa57 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.798277 7617 raft_consensus.cc:2454] T 796c7f3cc1f9426daecc89add3c6aa57 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:27:10.866834 4291 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:10.873039 7484 raft_consensus.cc:1261] T 55466814dd2a4396a37aa37e93d9119f P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:10.873309 7349 raft_consensus.cc:1261] T 55466814dd2a4396a37aa37e93d9119f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:10.874166 8457 consensus_queue.cc:1035] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:10.874476 8496 consensus_queue.cc:1035] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:10.880343 4300 catalog_manager.cc:5576] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719) does not have the latest schema for tablet 55466814dd2a4396a37aa37e93d9119f (table rebalancer_dynamic_table [id=2cb60813e32144149690f4e87b2ad229]). Expected version 1 got 0
I20240502 21:27:10.880932 4291 catalog_manager.cc:5576] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913) does not have the latest schema for tablet 55466814dd2a4396a37aa37e93d9119f (table rebalancer_dynamic_table [id=2cb60813e32144149690f4e87b2ad229]). Expected version 1 got 0
I20240502 21:27:10.882437 8504 tablet.cc:1722] T 55466814dd2a4396a37aa37e93d9119f P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:10.884506 4291 catalog_manager.cc:5576] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913) does not have the latest schema for tablet 55466814dd2a4396a37aa37e93d9119f (table rebalancer_dynamic_table [id=2cb60813e32144149690f4e87b2ad229]). Expected version 1 got 0
I20240502 21:27:10.886166 8504 tablet.cc:1620] T 55466814dd2a4396a37aa37e93d9119f P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:10.895211 8503 tablet.cc:1722] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:10.895319 4300 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=2cb60813e32144149690f4e87b2ad229] alter complete (version 1)
I20240502 21:27:10.895783 8503 tablet.cc:1620] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:10.896837 8506 tablet.cc:1722] T 55466814dd2a4396a37aa37e93d9119f P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:10.897464 8506 tablet.cc:1620] T 55466814dd2a4396a37aa37e93d9119f P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:10.898877 4300 catalog_manager.cc:5576] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719) does not have the latest schema for tablet 55466814dd2a4396a37aa37e93d9119f (table rebalancer_dynamic_table [id=2cb60813e32144149690f4e87b2ad229]). Expected version 1 got 0
I20240502 21:27:10.952576 8435 raft_consensus.cc:482] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:10.952752 8435 raft_consensus.cc:504] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:10.953595 8435 leader_election.cc:288] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:10.953963 7617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80304ab48274408393c2e245477c3a78" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:10.954126 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80304ab48274408393c2e245477c3a78" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:10.954221 7617 raft_consensus.cc:2454] T 80304ab48274408393c2e245477c3a78 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:10.954349 7349 raft_consensus.cc:2454] T 80304ab48274408393c2e245477c3a78 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:10.954649 7418 leader_election.cc:302] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168; no voters:
I20240502 21:27:10.954994 8435 raft_consensus.cc:2781] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:10.955129 8435 raft_consensus.cc:482] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:10.955267 8435 raft_consensus.cc:3037] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.957890 8435 raft_consensus.cc:504] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:10.958482 8435 leader_election.cc:288] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Requested vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:10.958846 7617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80304ab48274408393c2e245477c3a78" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:10.959014 7617 raft_consensus.cc:3037] T 80304ab48274408393c2e245477c3a78 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.958961 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80304ab48274408393c2e245477c3a78" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:10.959136 7349 raft_consensus.cc:3037] T 80304ab48274408393c2e245477c3a78 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.960918 7617 raft_consensus.cc:2454] T 80304ab48274408393c2e245477c3a78 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:10.961287 7418 leader_election.cc:302] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168; no voters:
I20240502 21:27:10.961609 8435 raft_consensus.cc:2781] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:10.961616 7349 raft_consensus.cc:2454] T 80304ab48274408393c2e245477c3a78 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:10.961779 8435 raft_consensus.cc:686] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Becoming Leader. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Running, Role: LEADER
I20240502 21:27:10.962056 8435 consensus_queue.cc:237] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:10.965289 4300 catalog_manager.cc:5484] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a72286397c4edc9a63eca3412be168 (127.29.82.71). New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:10.969050 8435 raft_consensus.cc:482] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:10.969177 8435 raft_consensus.cc:504] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:10.969710 8435 leader_election.cc:288] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:10.970057 7617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a85e7872c28a410496f31ecff9c71edb" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:10.970187 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a85e7872c28a410496f31ecff9c71edb" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:27:10.970273 7617 raft_consensus.cc:2454] T a85e7872c28a410496f31ecff9c71edb P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:10.970417 4443 raft_consensus.cc:2454] T a85e7872c28a410496f31ecff9c71edb P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:10.970702 7418 leader_election.cc:302] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168; no voters:
I20240502 21:27:10.970965 8435 raft_consensus.cc:2781] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:10.971058 8435 raft_consensus.cc:482] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:10.971164 8435 raft_consensus.cc:3037] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.973310 8435 raft_consensus.cc:504] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:10.973891 8435 leader_election.cc:288] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Requested vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:10.974313 7617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a85e7872c28a410496f31ecff9c71edb" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:10.974316 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a85e7872c28a410496f31ecff9c71edb" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:27:10.974525 7617 raft_consensus.cc:3037] T a85e7872c28a410496f31ecff9c71edb P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.974730 4443 raft_consensus.cc:3037] T a85e7872c28a410496f31ecff9c71edb P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.977468 7617 raft_consensus.cc:2454] T a85e7872c28a410496f31ecff9c71edb P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:10.977499 4443 raft_consensus.cc:2454] T a85e7872c28a410496f31ecff9c71edb P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:10.977823 7418 leader_election.cc:302] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168; no voters:
I20240502 21:27:10.978142 8435 raft_consensus.cc:2781] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:10.978303 8435 raft_consensus.cc:686] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Becoming Leader. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Running, Role: LEADER
I20240502 21:27:10.978550 8435 consensus_queue.cc:237] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:10.981485 4300 catalog_manager.cc:5484] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a72286397c4edc9a63eca3412be168 (127.29.82.71). New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:10.999119 4300 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:11.008544 8504 tablet.cc:1722] T 55466814dd2a4396a37aa37e93d9119f P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:11.008848 8503 tablet.cc:1722] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:11.009182 8504 tablet.cc:1620] T 55466814dd2a4396a37aa37e93d9119f P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:11.009282 8503 tablet.cc:1620] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:11.011296 8506 tablet.cc:1722] T 55466814dd2a4396a37aa37e93d9119f P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:11.011780 8506 tablet.cc:1620] T 55466814dd2a4396a37aa37e93d9119f P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:11.021502 4298 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=2cb60813e32144149690f4e87b2ad229] alter complete (version 2)
I20240502 21:27:11.126116 4300 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:11.133821 8503 tablet.cc:1722] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:11.135119 8506 tablet.cc:1722] T 55466814dd2a4396a37aa37e93d9119f P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:11.135622 8504 tablet.cc:1722] T 55466814dd2a4396a37aa37e93d9119f P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:11.144313 4300 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=2cb60813e32144149690f4e87b2ad229] alter complete (version 3)
I20240502 21:27:11.188539 8425 consensus_queue.cc:1035] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:11.193018 8180 consensus_queue.cc:1035] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:11.247318 4300 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:11.248040 4300 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:11.250892 4300 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 55466814dd2a4396a37aa37e93d9119f
I20240502 21:27:11.251672 7195 tablet_service.cc:1514] Processing DeleteTablet for tablet 55466814dd2a4396a37aa37e93d9119f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:11 UTC) from {username='slave'} at 127.0.0.1:44652
I20240502 21:27:11.251958 7329 tablet_service.cc:1514] Processing DeleteTablet for tablet 55466814dd2a4396a37aa37e93d9119f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:11 UTC) from {username='slave'} at 127.0.0.1:47078
I20240502 21:27:11.252307 7462 tablet_service.cc:1514] Processing DeleteTablet for tablet 55466814dd2a4396a37aa37e93d9119f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:11 UTC) from {username='slave'} at 127.0.0.1:37182
I20240502 21:27:11.262385 8514 tablet_replica.cc:330] T 55466814dd2a4396a37aa37e93d9119f P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:27:11.262638 8515 tablet_replica.cc:330] T 55466814dd2a4396a37aa37e93d9119f P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:27:11.262701 8514 raft_consensus.cc:2229] T 55466814dd2a4396a37aa37e93d9119f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:11.262784 8513 tablet_replica.cc:330] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:27:11.262881 8515 raft_consensus.cc:2229] T 55466814dd2a4396a37aa37e93d9119f P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:11.263002 8513 raft_consensus.cc:2229] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:11.262961 8514 raft_consensus.cc:2258] T 55466814dd2a4396a37aa37e93d9119f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:11.263108 8515 raft_consensus.cc:2258] T 55466814dd2a4396a37aa37e93d9119f P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:11.263803 8513 raft_consensus.cc:2258] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:11.264024 8515 ts_tablet_manager.cc:1875] T 55466814dd2a4396a37aa37e93d9119f P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:11.264042 8514 ts_tablet_manager.cc:1875] T 55466814dd2a4396a37aa37e93d9119f P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:11.264972 8513 ts_tablet_manager.cc:1875] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:11.270424 8514 ts_tablet_manager.cc:1888] T 55466814dd2a4396a37aa37e93d9119f P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:11.270634 8514 log.cc:1196] T 55466814dd2a4396a37aa37e93d9119f P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/55466814dd2a4396a37aa37e93d9119f
I20240502 21:27:11.271245 8514 ts_tablet_manager.cc:1909] T 55466814dd2a4396a37aa37e93d9119f P d0456c5e55a542128688904e003d403c: Deleting consensus metadata
I20240502 21:27:11.272408 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet 55466814dd2a4396a37aa37e93d9119f (table rebalancer_dynamic_table_new_name [id=2cb60813e32144149690f4e87b2ad229]) successfully deleted
I20240502 21:27:11.272941 8513 ts_tablet_manager.cc:1888] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:11.273052 8513 log.cc:1196] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/55466814dd2a4396a37aa37e93d9119f
I20240502 21:27:11.273312 8515 ts_tablet_manager.cc:1888] T 55466814dd2a4396a37aa37e93d9119f P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:11.273468 8513 ts_tablet_manager.cc:1909] T 55466814dd2a4396a37aa37e93d9119f P 39981a8dfe94462b9a73ba489af4c714: Deleting consensus metadata
I20240502 21:27:11.273490 8515 log.cc:1196] T 55466814dd2a4396a37aa37e93d9119f P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/55466814dd2a4396a37aa37e93d9119f
I20240502 21:27:11.274042 8515 ts_tablet_manager.cc:1909] T 55466814dd2a4396a37aa37e93d9119f P 31a72286397c4edc9a63eca3412be168: Deleting consensus metadata
I20240502 21:27:11.274349 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet 55466814dd2a4396a37aa37e93d9119f (table rebalancer_dynamic_table_new_name [id=2cb60813e32144149690f4e87b2ad229]) successfully deleted
I20240502 21:27:11.275373 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet 55466814dd2a4396a37aa37e93d9119f (table rebalancer_dynamic_table_new_name [id=2cb60813e32144149690f4e87b2ad229]) successfully deleted
I20240502 21:27:11.353720 4300 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:11.361881 7462 tablet_service.cc:1467] Processing CreateTablet for tablet 85ec23a4bb614cb69837a465c67966d6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3304cb517d0c4c80be39beea3b5e30fc]), partition=
I20240502 21:27:11.362329 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85ec23a4bb614cb69837a465c67966d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:11.362670 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 85ec23a4bb614cb69837a465c67966d6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3304cb517d0c4c80be39beea3b5e30fc]), partition=
I20240502 21:27:11.363018 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85ec23a4bb614cb69837a465c67966d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:11.367352 4818 tablet_service.cc:1467] Processing CreateTablet for tablet 85ec23a4bb614cb69837a465c67966d6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3304cb517d0c4c80be39beea3b5e30fc]), partition=
I20240502 21:27:11.367725 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85ec23a4bb614cb69837a465c67966d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:11.374614 8519 tablet_bootstrap.cc:492] T 85ec23a4bb614cb69837a465c67966d6 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:11.375056 8517 tablet_bootstrap.cc:492] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:11.375548 8518 tablet_bootstrap.cc:492] T 85ec23a4bb614cb69837a465c67966d6 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:11.377846 8518 tablet_bootstrap.cc:654] T 85ec23a4bb614cb69837a465c67966d6 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:11.377889 8517 tablet_bootstrap.cc:654] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:11.378645 8519 tablet_bootstrap.cc:654] T 85ec23a4bb614cb69837a465c67966d6 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:11.380932 8519 tablet_bootstrap.cc:492] T 85ec23a4bb614cb69837a465c67966d6 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:11.381083 8519 ts_tablet_manager.cc:1367] T 85ec23a4bb614cb69837a465c67966d6 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 21:27:11.381227 8518 tablet_bootstrap.cc:492] T 85ec23a4bb614cb69837a465c67966d6 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:27:11.381561 8518 ts_tablet_manager.cc:1367] T 85ec23a4bb614cb69837a465c67966d6 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240502 21:27:11.381875 8519 raft_consensus.cc:348] T 85ec23a4bb614cb69837a465c67966d6 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:11.382287 8519 raft_consensus.cc:374] T 85ec23a4bb614cb69837a465c67966d6 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:11.382402 8519 raft_consensus.cc:729] T 85ec23a4bb614cb69837a465c67966d6 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:11.382393 8518 raft_consensus.cc:348] T 85ec23a4bb614cb69837a465c67966d6 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:11.382609 8518 raft_consensus.cc:374] T 85ec23a4bb614cb69837a465c67966d6 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:11.382687 8518 raft_consensus.cc:729] T 85ec23a4bb614cb69837a465c67966d6 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:11.382629 8519 consensus_queue.cc:260] T 85ec23a4bb614cb69837a465c67966d6 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:11.382946 8518 consensus_queue.cc:260] T 85ec23a4bb614cb69837a465c67966d6 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:11.383155 8519 ts_tablet_manager.cc:1398] T 85ec23a4bb614cb69837a465c67966d6 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:11.383600 8518 ts_tablet_manager.cc:1398] T 85ec23a4bb614cb69837a465c67966d6 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:11.391552 8517 tablet_bootstrap.cc:492] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:11.391765 8517 ts_tablet_manager.cc:1367] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.001s
I20240502 21:27:11.392390 8517 raft_consensus.cc:348] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:11.392565 8517 raft_consensus.cc:374] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:11.392648 8517 raft_consensus.cc:729] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:11.392855 8517 consensus_queue.cc:260] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:11.393362 8517 ts_tablet_manager.cc:1398] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:27:11.407722 8435 raft_consensus.cc:482] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:11.407882 8435 raft_consensus.cc:504] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:11.408625 8435 leader_election.cc:288] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:27:11.409211 4704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85ec23a4bb614cb69837a465c67966d6" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:27:11.409469 4704 raft_consensus.cc:2454] T 85ec23a4bb614cb69837a465c67966d6 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:11.409965 7420 leader_election.cc:302] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 398c3c8f3195492c900439ad5e1931bb; no voters:
I20240502 21:27:11.410313 8435 raft_consensus.cc:2781] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:11.410418 8435 raft_consensus.cc:482] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:11.410496 8435 raft_consensus.cc:3037] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:11.413504 8435 raft_consensus.cc:504] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:11.414142 8435 leader_election.cc:288] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Requested vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:27:11.414743 4704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85ec23a4bb614cb69837a465c67966d6" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:27:11.414937 4704 raft_consensus.cc:3037] T 85ec23a4bb614cb69837a465c67966d6 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:11.417057 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85ec23a4bb614cb69837a465c67966d6" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:27:11.417178 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85ec23a4bb614cb69837a465c67966d6" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:27:11.417371 4838 raft_consensus.cc:2454] T 85ec23a4bb614cb69837a465c67966d6 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:11.417776 4704 raft_consensus.cc:2454] T 85ec23a4bb614cb69837a465c67966d6 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:11.418589 7420 leader_election.cc:302] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 398c3c8f3195492c900439ad5e1931bb; no voters: 73a1f2ca1f544de9b0e3b37015f85328
I20240502 21:27:11.418910 8466 raft_consensus.cc:2781] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:11.419059 8466 raft_consensus.cc:686] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Becoming Leader. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Running, Role: LEADER
I20240502 21:27:11.419389 8466 consensus_queue.cc:237] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:11.421129 8435 consensus_queue.cc:1035] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:11.424166 4298 catalog_manager.cc:5484] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a72286397c4edc9a63eca3412be168 (127.29.82.71). New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:11.435873 8526 consensus_queue.cc:1035] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:11.522838 4298 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:11.531158 4298 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:11.532848 7462 tablet_service.cc:1467] Processing CreateTablet for tablet 9e5434cb329e4bea85317d18f6e6867d (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=37d345f972f34b419de6c03d44f832d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:11.533166 7195 tablet_service.cc:1467] Processing CreateTablet for tablet 9e5434cb329e4bea85317d18f6e6867d (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=37d345f972f34b419de6c03d44f832d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:11.533443 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e5434cb329e4bea85317d18f6e6867d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:11.533687 7195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e5434cb329e4bea85317d18f6e6867d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:11.535100 7595 tablet_service.cc:1467] Processing CreateTablet for tablet 9e5434cb329e4bea85317d18f6e6867d (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=37d345f972f34b419de6c03d44f832d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:11.535130 7596 tablet_service.cc:1467] Processing CreateTablet for tablet 29235795acc147f68d2ccce93564ebd3 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=37d345f972f34b419de6c03d44f832d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:11.535715 7595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e5434cb329e4bea85317d18f6e6867d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:11.535986 7596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29235795acc147f68d2ccce93564ebd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:11.536720 7193 tablet_service.cc:1467] Processing CreateTablet for tablet e6f70f1bafd44613abb202a37d4c8b64 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=37d345f972f34b419de6c03d44f832d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:11.537106 7193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6f70f1bafd44613abb202a37d4c8b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:11.540035 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 29235795acc147f68d2ccce93564ebd3 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=37d345f972f34b419de6c03d44f832d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:11.540524 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29235795acc147f68d2ccce93564ebd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:11.542147 4818 tablet_service.cc:1467] Processing CreateTablet for tablet e6f70f1bafd44613abb202a37d4c8b64 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=37d345f972f34b419de6c03d44f832d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:11.542414 7463 tablet_service.cc:1467] Processing CreateTablet for tablet 29235795acc147f68d2ccce93564ebd3 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=37d345f972f34b419de6c03d44f832d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:11.542634 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6f70f1bafd44613abb202a37d4c8b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:11.546550 7464 tablet_service.cc:1467] Processing CreateTablet for tablet e6f70f1bafd44613abb202a37d4c8b64 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=37d345f972f34b419de6c03d44f832d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:11.548307 8466 consensus_queue.cc:1035] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:11.554481 8518 tablet_bootstrap.cc:492] T 29235795acc147f68d2ccce93564ebd3 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:11.556619 8518 tablet_bootstrap.cc:654] T 29235795acc147f68d2ccce93564ebd3 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:11.559387 8518 tablet_bootstrap.cc:492] T 29235795acc147f68d2ccce93564ebd3 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:27:11.559695 8518 ts_tablet_manager.cc:1367] T 29235795acc147f68d2ccce93564ebd3 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240502 21:27:11.560400 8519 tablet_bootstrap.cc:492] T e6f70f1bafd44613abb202a37d4c8b64 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:11.561282 7463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29235795acc147f68d2ccce93564ebd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:11.562068 8519 tablet_bootstrap.cc:654] T e6f70f1bafd44613abb202a37d4c8b64 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:11.562894 8518 raft_consensus.cc:348] T 29235795acc147f68d2ccce93564ebd3 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:11.563216 8518 raft_consensus.cc:374] T 29235795acc147f68d2ccce93564ebd3 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:11.563299 8518 raft_consensus.cc:729] T 29235795acc147f68d2ccce93564ebd3 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:11.563616 8518 consensus_queue.cc:260] T 29235795acc147f68d2ccce93564ebd3 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:11.563831 7464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6f70f1bafd44613abb202a37d4c8b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:11.564257 8518 ts_tablet_manager.cc:1398] T 29235795acc147f68d2ccce93564ebd3 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 21:27:11.565465 8532 tablet_bootstrap.cc:492] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:11.567768 8532 tablet_bootstrap.cc:654] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:11.573982 4705 raft_consensus.cc:1261] T 85ec23a4bb614cb69837a465c67966d6 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:11.574334 4839 raft_consensus.cc:3037] T 85ec23a4bb614cb69837a465c67966d6 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:11.575536 8466 consensus_queue.cc:1035] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:11.577302 4839 raft_consensus.cc:1261] T 85ec23a4bb614cb69837a465c67966d6 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:11.577946 8466 consensus_queue.cc:1035] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:11.582934 8519 tablet_bootstrap.cc:492] T e6f70f1bafd44613abb202a37d4c8b64 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:11.583410 8519 ts_tablet_manager.cc:1367] T e6f70f1bafd44613abb202a37d4c8b64 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.000s
I20240502 21:27:11.585657 8525 consensus_queue.cc:1035] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:11.592929 8532 tablet_bootstrap.cc:492] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:11.593355 8519 raft_consensus.cc:348] T e6f70f1bafd44613abb202a37d4c8b64 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:11.593578 8532 ts_tablet_manager.cc:1367] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.015s
I20240502 21:27:11.594861 8532 raft_consensus.cc:348] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:11.595167 8532 raft_consensus.cc:374] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:11.595362 8532 raft_consensus.cc:729] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:11.595903 8532 consensus_queue.cc:260] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:11.596037 8519 raft_consensus.cc:374] T e6f70f1bafd44613abb202a37d4c8b64 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:11.596144 8519 raft_consensus.cc:729] T e6f70f1bafd44613abb202a37d4c8b64 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:11.596896 8519 consensus_queue.cc:260] T e6f70f1bafd44613abb202a37d4c8b64 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:11.598793 8519 ts_tablet_manager.cc:1398] T e6f70f1bafd44613abb202a37d4c8b64 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240502 21:27:11.606555 8531 tablet_bootstrap.cc:492] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:11.606858 8532 ts_tablet_manager.cc:1398] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240502 21:27:11.609524 8532 tablet_bootstrap.cc:492] T 9e5434cb329e4bea85317d18f6e6867d P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:11.609715 8517 tablet_bootstrap.cc:492] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:11.611718 8517 tablet_bootstrap.cc:654] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:11.614641 8531 tablet_bootstrap.cc:654] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:11.618559 8532 tablet_bootstrap.cc:654] T 9e5434cb329e4bea85317d18f6e6867d P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:11.621140 8517 tablet_bootstrap.cc:492] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:11.621414 8517 ts_tablet_manager.cc:1367] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.006s
I20240502 21:27:11.621734 4291 catalog_manager.cc:5576] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449) does not have the latest schema for tablet 85ec23a4bb614cb69837a465c67966d6 (table rebalancer_dynamic_table [id=3304cb517d0c4c80be39beea3b5e30fc]). Expected version 1 got 0
I20240502 21:27:11.622548 8517 raft_consensus.cc:348] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:11.622995 8517 raft_consensus.cc:374] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:11.623155 8517 raft_consensus.cc:729] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:11.623468 8517 consensus_queue.cc:260] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:11.623647 8504 tablet.cc:1722] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:11.624146 8504 tablet.cc:1620] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:11.624799 8517 ts_tablet_manager.cc:1398] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:11.631976 8531 tablet_bootstrap.cc:492] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:11.632155 8531 ts_tablet_manager.cc:1367] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20240502 21:27:11.633085 8531 raft_consensus.cc:348] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:11.633370 8531 raft_consensus.cc:374] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:11.633489 8531 raft_consensus.cc:729] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:11.633735 8531 consensus_queue.cc:260] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:11.636277 8531 ts_tablet_manager.cc:1398] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:27:11.637233 8531 tablet_bootstrap.cc:492] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:11.641359 8531 tablet_bootstrap.cc:654] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:11.642271 8548 tablet.cc:1722] T 85ec23a4bb614cb69837a465c67966d6 P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:11.644783 8548 tablet.cc:1620] T 85ec23a4bb614cb69837a465c67966d6 P 73a1f2ca1f544de9b0e3b37015f85328: MemRowSet was empty: no flush needed.
I20240502 21:27:11.646576 8549 tablet.cc:1722] T 85ec23a4bb614cb69837a465c67966d6 P 398c3c8f3195492c900439ad5e1931bb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:11.646848 8517 tablet_bootstrap.cc:492] T 9e5434cb329e4bea85317d18f6e6867d P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:11.647887 8549 tablet.cc:1620] T 85ec23a4bb614cb69837a465c67966d6 P 398c3c8f3195492c900439ad5e1931bb: MemRowSet was empty: no flush needed.
I20240502 21:27:11.648814 4298 catalog_manager.cc:5576] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449) does not have the latest schema for tablet 85ec23a4bb614cb69837a465c67966d6 (table rebalancer_dynamic_table [id=3304cb517d0c4c80be39beea3b5e30fc]). Expected version 1 got 0
I20240502 21:27:11.650244 8517 tablet_bootstrap.cc:654] T 9e5434cb329e4bea85317d18f6e6867d P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:11.651391 8531 tablet_bootstrap.cc:492] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:11.652047 8531 ts_tablet_manager.cc:1367] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.003s
I20240502 21:27:11.652959 8531 raft_consensus.cc:348] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:11.654314 8531 raft_consensus.cc:374] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:11.654456 8531 raft_consensus.cc:729] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:11.655175 8531 consensus_queue.cc:260] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:11.656778 4300 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=3304cb517d0c4c80be39beea3b5e30fc] alter complete (version 1)
I20240502 21:27:11.659451 8531 ts_tablet_manager.cc:1398] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 21:27:11.663194 8517 tablet_bootstrap.cc:492] T 9e5434cb329e4bea85317d18f6e6867d P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:11.663353 8517 ts_tablet_manager.cc:1367] T 9e5434cb329e4bea85317d18f6e6867d P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240502 21:27:11.665171 8532 tablet_bootstrap.cc:492] T 9e5434cb329e4bea85317d18f6e6867d P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:11.665341 8532 ts_tablet_manager.cc:1367] T 9e5434cb329e4bea85317d18f6e6867d P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.056s user 0.012s sys 0.006s
I20240502 21:27:11.665607 8517 raft_consensus.cc:348] T 9e5434cb329e4bea85317d18f6e6867d P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:11.665897 8517 raft_consensus.cc:374] T 9e5434cb329e4bea85317d18f6e6867d P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:11.666049 8517 raft_consensus.cc:729] T 9e5434cb329e4bea85317d18f6e6867d P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:11.666024 8532 raft_consensus.cc:348] T 9e5434cb329e4bea85317d18f6e6867d P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:11.666266 8532 raft_consensus.cc:374] T 9e5434cb329e4bea85317d18f6e6867d P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:11.666321 8532 raft_consensus.cc:729] T 9e5434cb329e4bea85317d18f6e6867d P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:11.666543 8532 consensus_queue.cc:260] T 9e5434cb329e4bea85317d18f6e6867d P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:11.667100 8532 ts_tablet_manager.cc:1398] T 9e5434cb329e4bea85317d18f6e6867d P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:11.666607 8517 consensus_queue.cc:260] T 9e5434cb329e4bea85317d18f6e6867d P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:11.669438 8517 ts_tablet_manager.cc:1398] T 9e5434cb329e4bea85317d18f6e6867d P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:27:11.669751 8517 tablet_bootstrap.cc:492] T e6f70f1bafd44613abb202a37d4c8b64 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:11.671273 8517 tablet_bootstrap.cc:654] T e6f70f1bafd44613abb202a37d4c8b64 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:11.673238 8517 tablet_bootstrap.cc:492] T e6f70f1bafd44613abb202a37d4c8b64 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:11.673430 8517 ts_tablet_manager.cc:1367] T e6f70f1bafd44613abb202a37d4c8b64 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:27:11.674016 8517 raft_consensus.cc:348] T e6f70f1bafd44613abb202a37d4c8b64 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:11.674189 8517 raft_consensus.cc:374] T e6f70f1bafd44613abb202a37d4c8b64 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:11.674264 8517 raft_consensus.cc:729] T e6f70f1bafd44613abb202a37d4c8b64 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:11.674495 8517 consensus_queue.cc:260] T e6f70f1bafd44613abb202a37d4c8b64 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:11.675117 8517 ts_tablet_manager.cc:1398] T e6f70f1bafd44613abb202a37d4c8b64 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:11.675726 7942 raft_consensus.cc:482] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:11.675885 7942 raft_consensus.cc:504] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:11.676627 7942 leader_election.cc:288] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:11.677106 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5434cb329e4bea85317d18f6e6867d" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:11.677325 7484 raft_consensus.cc:2454] T 9e5434cb329e4bea85317d18f6e6867d P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:11.677318 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5434cb329e4bea85317d18f6e6867d" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:11.677551 7215 raft_consensus.cc:2454] T 9e5434cb329e4bea85317d18f6e6867d P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:11.678043 7549 leader_election.cc:302] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:11.678462 7942 raft_consensus.cc:2781] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:11.678567 7942 raft_consensus.cc:482] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:11.678665 7942 raft_consensus.cc:3037] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:11.681422 7942 raft_consensus.cc:504] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:11.681986 7942 leader_election.cc:288] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Requested vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:11.682420 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5434cb329e4bea85317d18f6e6867d" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:11.682678 7484 raft_consensus.cc:3037] T 9e5434cb329e4bea85317d18f6e6867d P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:11.682659 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5434cb329e4bea85317d18f6e6867d" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:11.682883 7214 raft_consensus.cc:3037] T 9e5434cb329e4bea85317d18f6e6867d P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:11.684743 7484 raft_consensus.cc:2454] T 9e5434cb329e4bea85317d18f6e6867d P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:11.684968 7214 raft_consensus.cc:2454] T 9e5434cb329e4bea85317d18f6e6867d P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:11.685145 7549 leader_election.cc:302] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168; no voters:
I20240502 21:27:11.685544 7942 raft_consensus.cc:2781] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:11.685683 7942 raft_consensus.cc:686] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Becoming Leader. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Running, Role: LEADER
I20240502 21:27:11.686026 7942 consensus_queue.cc:237] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:11.689437 4294 catalog_manager.cc:5484] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ddd486109f5434cace13d60d633ac04 (127.29.82.72). New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:11.762820 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:11.770615 8504 tablet.cc:1722] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:11.771201 8504 tablet.cc:1620] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:11.772876 8549 tablet.cc:1722] T 85ec23a4bb614cb69837a465c67966d6 P 398c3c8f3195492c900439ad5e1931bb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:11.773769 8548 tablet.cc:1722] T 85ec23a4bb614cb69837a465c67966d6 P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:11.775164 8549 tablet.cc:1620] T 85ec23a4bb614cb69837a465c67966d6 P 398c3c8f3195492c900439ad5e1931bb: MemRowSet was empty: no flush needed.
I20240502 21:27:11.776165 8548 tablet.cc:1620] T 85ec23a4bb614cb69837a465c67966d6 P 73a1f2ca1f544de9b0e3b37015f85328: MemRowSet was empty: no flush needed.
I20240502 21:27:11.782181 4300 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=3304cb517d0c4c80be39beea3b5e30fc] alter complete (version 2)
I20240502 21:27:11.822881 7942 raft_consensus.cc:482] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:11.823065 7942 raft_consensus.cc:504] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:11.823760 7942 leader_election.cc:288] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:11.824702 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29235795acc147f68d2ccce93564ebd3" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:11.824986 7484 raft_consensus.cc:2454] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:11.825122 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29235795acc147f68d2ccce93564ebd3" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:27:11.825429 4705 raft_consensus.cc:2454] T 29235795acc147f68d2ccce93564ebd3 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:11.825708 7549 leader_election.cc:302] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168; no voters:
I20240502 21:27:11.827514 7942 raft_consensus.cc:2781] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:11.827630 7942 raft_consensus.cc:482] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:11.828188 7942 raft_consensus.cc:3037] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:11.830770 7942 raft_consensus.cc:504] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:11.831387 7942 leader_election.cc:288] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:11.831727 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29235795acc147f68d2ccce93564ebd3" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:27:11.831898 4705 raft_consensus.cc:3037] T 29235795acc147f68d2ccce93564ebd3 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:11.831841 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29235795acc147f68d2ccce93564ebd3" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:11.832049 7484 raft_consensus.cc:3037] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:11.834215 7484 raft_consensus.cc:2454] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:11.834574 7549 leader_election.cc:302] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168; no voters:
I20240502 21:27:11.834645 4705 raft_consensus.cc:2454] T 29235795acc147f68d2ccce93564ebd3 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:11.835134 7942 raft_consensus.cc:2781] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:11.835300 7942 raft_consensus.cc:686] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Becoming Leader. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Running, Role: LEADER
I20240502 21:27:11.835563 7942 consensus_queue.cc:237] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:11.839111 4294 catalog_manager.cc:5484] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ddd486109f5434cace13d60d633ac04 (127.29.82.72). New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:27:11.842283 8457 raft_consensus.cc:482] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:11.842439 8457 raft_consensus.cc:504] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:11.843112 8457 leader_election.cc:288] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:11.843745 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6f70f1bafd44613abb202a37d4c8b64" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:11.843786 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6f70f1bafd44613abb202a37d4c8b64" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:27:11.844015 4839 raft_consensus.cc:2454] T e6f70f1bafd44613abb202a37d4c8b64 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:11.844015 7484 raft_consensus.cc:2454] T e6f70f1bafd44613abb202a37d4c8b64 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:11.844463 7148 leader_election.cc:302] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:11.844749 8457 raft_consensus.cc:2781] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:11.844856 8457 raft_consensus.cc:482] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:11.844962 8457 raft_consensus.cc:3037] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:11.846731 8457 raft_consensus.cc:504] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:11.847309 8457 leader_election.cc:288] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:11.848141 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6f70f1bafd44613abb202a37d4c8b64" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:27:11.848273 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6f70f1bafd44613abb202a37d4c8b64" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:11.848361 4839 raft_consensus.cc:3037] T e6f70f1bafd44613abb202a37d4c8b64 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:11.848459 7484 raft_consensus.cc:3037] T e6f70f1bafd44613abb202a37d4c8b64 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:11.851171 4839 raft_consensus.cc:2454] T e6f70f1bafd44613abb202a37d4c8b64 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:11.851888 7148 leader_election.cc:302] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:11.852138 8457 raft_consensus.cc:2781] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:11.852604 8457 raft_consensus.cc:686] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:27:11.853212 8457 consensus_queue.cc:237] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:11.854408 7484 raft_consensus.cc:2454] T e6f70f1bafd44613abb202a37d4c8b64 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:11.874228 4294 catalog_manager.cc:5484] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69). New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:27:11.885288 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:11.898819 8504 tablet.cc:1722] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:11.899516 8548 tablet.cc:1722] T 85ec23a4bb614cb69837a465c67966d6 P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:11.909534 4294 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=3304cb517d0c4c80be39beea3b5e30fc] alter complete (version 3)
I20240502 21:27:11.910842 8549 tablet.cc:1722] T 85ec23a4bb614cb69837a465c67966d6 P 398c3c8f3195492c900439ad5e1931bb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:12.017845 4294 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:12.018133 4294 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:12.021420 4294 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 85ec23a4bb614cb69837a465c67966d6
I20240502 21:27:12.022248 7464 tablet_service.cc:1514] Processing DeleteTablet for tablet 85ec23a4bb614cb69837a465c67966d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:12 UTC) from {username='slave'} at 127.0.0.1:37182
I20240502 21:27:12.022469 4818 tablet_service.cc:1514] Processing DeleteTablet for tablet 85ec23a4bb614cb69837a465c67966d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:12 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:27:12.023057 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet 85ec23a4bb614cb69837a465c67966d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:12 UTC) from {username='slave'} at 127.0.0.1:44764
I20240502 21:27:12.033878 8560 tablet_replica.cc:330] T 85ec23a4bb614cb69837a465c67966d6 P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:27:12.034162 8559 tablet_replica.cc:330] T 85ec23a4bb614cb69837a465c67966d6 P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:27:12.034410 8560 raft_consensus.cc:2229] T 85ec23a4bb614cb69837a465c67966d6 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:12.034621 8559 raft_consensus.cc:2229] T 85ec23a4bb614cb69837a465c67966d6 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:12.034749 8560 raft_consensus.cc:2258] T 85ec23a4bb614cb69837a465c67966d6 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:12.034988 8559 raft_consensus.cc:2258] T 85ec23a4bb614cb69837a465c67966d6 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:12.035041 8558 tablet_replica.cc:330] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:27:12.035259 8558 raft_consensus.cc:2229] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:12.035734 8558 raft_consensus.cc:2258] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:12.035892 8560 ts_tablet_manager.cc:1875] T 85ec23a4bb614cb69837a465c67966d6 P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:12.036651 8559 ts_tablet_manager.cc:1875] T 85ec23a4bb614cb69837a465c67966d6 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:12.036773 8558 ts_tablet_manager.cc:1875] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:12.041316 8559 ts_tablet_manager.cc:1888] T 85ec23a4bb614cb69837a465c67966d6 P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:12.041472 8559 log.cc:1196] T 85ec23a4bb614cb69837a465c67966d6 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/85ec23a4bb614cb69837a465c67966d6
I20240502 21:27:12.041945 8559 ts_tablet_manager.cc:1909] T 85ec23a4bb614cb69837a465c67966d6 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:27:12.042686 8558 ts_tablet_manager.cc:1888] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:12.042799 8558 log.cc:1196] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/85ec23a4bb614cb69837a465c67966d6
I20240502 21:27:12.043061 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet 85ec23a4bb614cb69837a465c67966d6 (table rebalancer_dynamic_table_new_name [id=3304cb517d0c4c80be39beea3b5e30fc]) successfully deleted
I20240502 21:27:12.043232 8558 ts_tablet_manager.cc:1909] T 85ec23a4bb614cb69837a465c67966d6 P 31a72286397c4edc9a63eca3412be168: Deleting consensus metadata
I20240502 21:27:12.043911 8560 ts_tablet_manager.cc:1888] T 85ec23a4bb614cb69837a465c67966d6 P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:12.044027 8560 log.cc:1196] T 85ec23a4bb614cb69837a465c67966d6 P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/85ec23a4bb614cb69837a465c67966d6
I20240502 21:27:12.044154 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet 85ec23a4bb614cb69837a465c67966d6 (table rebalancer_dynamic_table_new_name [id=3304cb517d0c4c80be39beea3b5e30fc]) successfully deleted
I20240502 21:27:12.044497 8560 ts_tablet_manager.cc:1909] T 85ec23a4bb614cb69837a465c67966d6 P 398c3c8f3195492c900439ad5e1931bb: Deleting consensus metadata
I20240502 21:27:12.045799 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet 85ec23a4bb614cb69837a465c67966d6 (table rebalancer_dynamic_table_new_name [id=3304cb517d0c4c80be39beea3b5e30fc]) successfully deleted
I20240502 21:27:12.124377 4294 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:12.131928 7193 tablet_service.cc:1467] Processing CreateTablet for tablet a31bfb0283164a4da34b6af1bc5deb90 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bcd2a277a73646e598a0f6490ce410c1]), partition=
I20240502 21:27:12.132371 7193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a31bfb0283164a4da34b6af1bc5deb90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:12.132390 4818 tablet_service.cc:1467] Processing CreateTablet for tablet a31bfb0283164a4da34b6af1bc5deb90 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bcd2a277a73646e598a0f6490ce410c1]), partition=
I20240502 21:27:12.132755 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a31bfb0283164a4da34b6af1bc5deb90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:12.138181 8562 tablet_bootstrap.cc:492] T a31bfb0283164a4da34b6af1bc5deb90 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:12.138286 8532 tablet_bootstrap.cc:492] T a31bfb0283164a4da34b6af1bc5deb90 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:12.139910 8532 tablet_bootstrap.cc:654] T a31bfb0283164a4da34b6af1bc5deb90 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:12.140398 8562 tablet_bootstrap.cc:654] T a31bfb0283164a4da34b6af1bc5deb90 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:12.141160 7329 tablet_service.cc:1467] Processing CreateTablet for tablet a31bfb0283164a4da34b6af1bc5deb90 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bcd2a277a73646e598a0f6490ce410c1]), partition=
I20240502 21:27:12.141567 7329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a31bfb0283164a4da34b6af1bc5deb90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:12.143184 8562 tablet_bootstrap.cc:492] T a31bfb0283164a4da34b6af1bc5deb90 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:12.143309 8562 ts_tablet_manager.cc:1367] T a31bfb0283164a4da34b6af1bc5deb90 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:27:12.144028 8562 raft_consensus.cc:348] T a31bfb0283164a4da34b6af1bc5deb90 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:12.144351 8562 raft_consensus.cc:374] T a31bfb0283164a4da34b6af1bc5deb90 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:12.144503 8562 raft_consensus.cc:729] T a31bfb0283164a4da34b6af1bc5deb90 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:12.144892 8562 consensus_queue.cc:260] T a31bfb0283164a4da34b6af1bc5deb90 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:12.145570 8562 ts_tablet_manager.cc:1398] T a31bfb0283164a4da34b6af1bc5deb90 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:12.147262 8532 tablet_bootstrap.cc:492] T a31bfb0283164a4da34b6af1bc5deb90 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:12.147406 8532 ts_tablet_manager.cc:1367] T a31bfb0283164a4da34b6af1bc5deb90 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240502 21:27:12.148239 8532 raft_consensus.cc:348] T a31bfb0283164a4da34b6af1bc5deb90 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:12.148465 8532 raft_consensus.cc:374] T a31bfb0283164a4da34b6af1bc5deb90 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:12.148567 8532 raft_consensus.cc:729] T a31bfb0283164a4da34b6af1bc5deb90 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:12.149031 8532 consensus_queue.cc:260] T a31bfb0283164a4da34b6af1bc5deb90 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:12.149631 8532 ts_tablet_manager.cc:1398] T a31bfb0283164a4da34b6af1bc5deb90 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:12.151007 8565 tablet_bootstrap.cc:492] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:12.154067 8565 tablet_bootstrap.cc:654] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:12.156293 8565 tablet_bootstrap.cc:492] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:12.156422 8565 ts_tablet_manager.cc:1367] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:27:12.157035 8565 raft_consensus.cc:348] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:12.157224 8565 raft_consensus.cc:374] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:12.157330 8565 raft_consensus.cc:729] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:12.157564 8565 consensus_queue.cc:260] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:12.158182 8565 ts_tablet_manager.cc:1398] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:12.175702 7942 consensus_queue.cc:1035] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:12.180189 7942 consensus_queue.cc:1035] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:12.247829 8457 consensus_queue.cc:1035] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:12.252154 8457 consensus_queue.cc:1035] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:12.289021 7942 consensus_queue.cc:1035] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:12.294332 7942 consensus_queue.cc:1035] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:12.339408 8574 raft_consensus.cc:482] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:12.339609 8574 raft_consensus.cc:504] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:12.340288 8574 leader_election.cc:288] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:12.340701 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a31bfb0283164a4da34b6af1bc5deb90" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:12.340776 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a31bfb0283164a4da34b6af1bc5deb90" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:27:12.340929 7214 raft_consensus.cc:2454] T a31bfb0283164a4da34b6af1bc5deb90 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 0.
I20240502 21:27:12.341029 4839 raft_consensus.cc:2454] T a31bfb0283164a4da34b6af1bc5deb90 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 0.
I20240502 21:27:12.341341 7283 leader_election.cc:302] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:12.341610 8574 raft_consensus.cc:2781] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:12.341728 8574 raft_consensus.cc:482] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:12.341804 8574 raft_consensus.cc:3037] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:12.344094 8574 raft_consensus.cc:504] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:12.344645 8574 leader_election.cc:288] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Requested vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:12.344972 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a31bfb0283164a4da34b6af1bc5deb90" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:12.345078 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a31bfb0283164a4da34b6af1bc5deb90" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:27:12.345167 7214 raft_consensus.cc:3037] T a31bfb0283164a4da34b6af1bc5deb90 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:12.345283 4839 raft_consensus.cc:3037] T a31bfb0283164a4da34b6af1bc5deb90 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:12.347126 7214 raft_consensus.cc:2454] T a31bfb0283164a4da34b6af1bc5deb90 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
I20240502 21:27:12.347143 4839 raft_consensus.cc:2454] T a31bfb0283164a4da34b6af1bc5deb90 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
I20240502 21:27:12.347532 7283 leader_election.cc:302] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:12.347839 8574 raft_consensus.cc:2781] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:12.348001 8574 raft_consensus.cc:686] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Becoming Leader. State: Replica: d0456c5e55a542128688904e003d403c, State: Running, Role: LEADER
I20240502 21:27:12.348294 8574 consensus_queue.cc:237] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:12.351455 4294 catalog_manager.cc:5484] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c reported cstate change: term changed from 0 to 1, leader changed from <none> to d0456c5e55a542128688904e003d403c (127.29.82.70). New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:12.449834 4294 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:12.460877 7329 tablet_service.cc:1467] Processing CreateTablet for tablet 61e18d2b13644f6a99225c106c594914 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b87f68aedb34492aade9e648ad016b2c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:12.461340 7329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61e18d2b13644f6a99225c106c594914. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:12.464411 7595 tablet_service.cc:1467] Processing CreateTablet for tablet 61e18d2b13644f6a99225c106c594914 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b87f68aedb34492aade9e648ad016b2c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:12.464910 7595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61e18d2b13644f6a99225c106c594914. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:12.465763 7195 tablet_service.cc:1467] Processing CreateTablet for tablet a5d47492f64f49d584246fc5007281aa (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b87f68aedb34492aade9e648ad016b2c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:12.466168 7195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5d47492f64f49d584246fc5007281aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:12.467603 7596 tablet_service.cc:1467] Processing CreateTablet for tablet a5d47492f64f49d584246fc5007281aa (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b87f68aedb34492aade9e648ad016b2c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:12.467990 7596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5d47492f64f49d584246fc5007281aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:12.467964 7193 tablet_service.cc:1467] Processing CreateTablet for tablet 61e18d2b13644f6a99225c106c594914 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b87f68aedb34492aade9e648ad016b2c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:12.467974 7328 tablet_service.cc:1467] Processing CreateTablet for tablet 3394f5f087304842ada4212f2e524356 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b87f68aedb34492aade9e648ad016b2c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:12.468348 7193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61e18d2b13644f6a99225c106c594914. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:12.468402 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3394f5f087304842ada4212f2e524356. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:12.471777 8565 tablet_bootstrap.cc:492] T 61e18d2b13644f6a99225c106c594914 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:12.471935 7597 tablet_service.cc:1467] Processing CreateTablet for tablet 3394f5f087304842ada4212f2e524356 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b87f68aedb34492aade9e648ad016b2c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:12.472319 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3394f5f087304842ada4212f2e524356. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:12.473415 8565 tablet_bootstrap.cc:654] T 61e18d2b13644f6a99225c106c594914 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:12.475147 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 3394f5f087304842ada4212f2e524356 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b87f68aedb34492aade9e648ad016b2c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:12.475654 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3394f5f087304842ada4212f2e524356. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:12.475780 8532 tablet_bootstrap.cc:492] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:12.477241 8532 tablet_bootstrap.cc:654] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:12.483777 8583 tablet_bootstrap.cc:492] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:12.484560 7464 tablet_service.cc:1467] Processing CreateTablet for tablet a5d47492f64f49d584246fc5007281aa (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b87f68aedb34492aade9e648ad016b2c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:12.484576 4292 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:12.485076 7464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5d47492f64f49d584246fc5007281aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:12.487016 8583 tablet_bootstrap.cc:654] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:12.488572 8565 tablet_bootstrap.cc:492] T 61e18d2b13644f6a99225c106c594914 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:12.488781 8565 ts_tablet_manager.cc:1367] T 61e18d2b13644f6a99225c106c594914 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240502 21:27:12.489673 8565 raft_consensus.cc:348] T 61e18d2b13644f6a99225c106c594914 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:12.489918 8565 raft_consensus.cc:374] T 61e18d2b13644f6a99225c106c594914 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:12.490008 8565 raft_consensus.cc:729] T 61e18d2b13644f6a99225c106c594914 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:12.490285 8565 consensus_queue.cc:260] T 61e18d2b13644f6a99225c106c594914 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:12.490867 8565 ts_tablet_manager.cc:1398] T 61e18d2b13644f6a99225c106c594914 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:12.493168 8565 tablet_bootstrap.cc:492] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:12.495044 8565 tablet_bootstrap.cc:654] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:12.495184 4300 catalog_manager.cc:5576] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719) does not have the latest schema for tablet a31bfb0283164a4da34b6af1bc5deb90 (table rebalancer_dynamic_table [id=bcd2a277a73646e598a0f6490ce410c1]). Expected version 1 got 0
I20240502 21:27:12.495532 7214 raft_consensus.cc:1261] T a31bfb0283164a4da34b6af1bc5deb90 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:12.497037 8574 consensus_queue.cc:1035] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:12.501572 8565 tablet_bootstrap.cc:492] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:12.501777 8565 ts_tablet_manager.cc:1367] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20240502 21:27:12.502836 8565 raft_consensus.cc:348] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:12.503089 8565 raft_consensus.cc:374] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:12.503068 4839 raft_consensus.cc:1261] T a31bfb0283164a4da34b6af1bc5deb90 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:12.503234 8565 raft_consensus.cc:729] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:12.503532 8565 consensus_queue.cc:260] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:12.504282 8472 consensus_queue.cc:1035] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:12.505278 8565 ts_tablet_manager.cc:1398] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:27:12.509400 4292 catalog_manager.cc:5576] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449) does not have the latest schema for tablet a31bfb0283164a4da34b6af1bc5deb90 (table rebalancer_dynamic_table [id=bcd2a277a73646e598a0f6490ce410c1]). Expected version 1 got 0
I20240502 21:27:12.510663 8532 tablet_bootstrap.cc:492] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:12.510893 8532 ts_tablet_manager.cc:1367] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.001s
I20240502 21:27:12.511840 8532 raft_consensus.cc:348] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:12.512125 8532 raft_consensus.cc:374] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:12.512213 8532 raft_consensus.cc:729] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:12.512501 8532 consensus_queue.cc:260] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:12.513180 8532 ts_tablet_manager.cc:1398] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:12.513312 4292 catalog_manager.cc:5576] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449) does not have the latest schema for tablet a31bfb0283164a4da34b6af1bc5deb90 (table rebalancer_dynamic_table [id=bcd2a277a73646e598a0f6490ce410c1]). Expected version 1 got 0
I20240502 21:27:12.513474 8532 tablet_bootstrap.cc:492] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:12.515512 8584 tablet_bootstrap.cc:492] T 3394f5f087304842ada4212f2e524356 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:12.515650 8532 tablet_bootstrap.cc:654] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:12.519418 8532 tablet_bootstrap.cc:492] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:12.519558 8532 ts_tablet_manager.cc:1367] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 21:27:12.519554 8472 raft_consensus.cc:482] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:12.519734 8472 raft_consensus.cc:504] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:12.520635 8472 leader_election.cc:288] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:27:12.522068 7617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3394f5f087304842ada4212f2e524356" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:12.522192 8532 raft_consensus.cc:348] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:12.522372 8532 raft_consensus.cc:374] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:12.522459 8532 raft_consensus.cc:729] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
W20240502 21:27:12.524693 7282 leader_election.cc:341] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:12.524832 8589 tablet_bootstrap.cc:492] T a5d47492f64f49d584246fc5007281aa P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:12.525207 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3394f5f087304842ada4212f2e524356" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
W20240502 21:27:12.525920 7285 leader_election.cc:341] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:12.526038 7285 leader_election.cc:302] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0456c5e55a542128688904e003d403c; no voters: 0ddd486109f5434cace13d60d633ac04, 398c3c8f3195492c900439ad5e1931bb
I20240502 21:27:12.526157 8592 tablet.cc:1722] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:12.526479 8532 consensus_queue.cc:260] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:12.526762 8472 raft_consensus.cc:2726] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:27:12.526822 8592 tablet.cc:1620] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:12.527016 8584 tablet_bootstrap.cc:654] T 3394f5f087304842ada4212f2e524356 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:12.528350 8532 ts_tablet_manager.cc:1398] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240502 21:27:12.534022 8589 tablet_bootstrap.cc:654] T a5d47492f64f49d584246fc5007281aa P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:12.535293 4292 catalog_manager.cc:5576] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193) does not have the latest schema for tablet a31bfb0283164a4da34b6af1bc5deb90 (table rebalancer_dynamic_table [id=bcd2a277a73646e598a0f6490ce410c1]). Expected version 1 got 0
I20240502 21:27:12.541038 8589 tablet_bootstrap.cc:492] T a5d47492f64f49d584246fc5007281aa P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:12.541230 8589 ts_tablet_manager.cc:1367] T a5d47492f64f49d584246fc5007281aa P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.000s
I20240502 21:27:12.541544 4294 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=bcd2a277a73646e598a0f6490ce410c1] alter complete (version 1)
I20240502 21:27:12.542268 8589 raft_consensus.cc:348] T a5d47492f64f49d584246fc5007281aa P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:12.542578 8589 raft_consensus.cc:374] T a5d47492f64f49d584246fc5007281aa P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:12.542404 8594 tablet.cc:1722] T a31bfb0283164a4da34b6af1bc5deb90 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:12.542759 8589 raft_consensus.cc:729] T a5d47492f64f49d584246fc5007281aa P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:12.543033 8589 consensus_queue.cc:260] T a5d47492f64f49d584246fc5007281aa P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:12.543239 8594 tablet.cc:1620] T a31bfb0283164a4da34b6af1bc5deb90 P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:12.544842 8589 ts_tablet_manager.cc:1398] T a5d47492f64f49d584246fc5007281aa P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:12.546510 4292 catalog_manager.cc:5576] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193) does not have the latest schema for tablet a31bfb0283164a4da34b6af1bc5deb90 (table rebalancer_dynamic_table [id=bcd2a277a73646e598a0f6490ce410c1]). Expected version 1 got 0
I20240502 21:27:12.547703 8584 tablet_bootstrap.cc:492] T 3394f5f087304842ada4212f2e524356 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:12.550024 8583 tablet_bootstrap.cc:492] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:27:12.550444 8584 ts_tablet_manager.cc:1367] T 3394f5f087304842ada4212f2e524356 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.002s
I20240502 21:27:12.551101 8584 raft_consensus.cc:348] T 3394f5f087304842ada4212f2e524356 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:12.551273 8584 raft_consensus.cc:374] T 3394f5f087304842ada4212f2e524356 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:12.551370 8584 raft_consensus.cc:729] T 3394f5f087304842ada4212f2e524356 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:12.551609 8583 ts_tablet_manager.cc:1367] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.068s user 0.010s sys 0.000s
I20240502 21:27:12.551599 8584 consensus_queue.cc:260] T 3394f5f087304842ada4212f2e524356 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:12.552098 8584 ts_tablet_manager.cc:1398] T 3394f5f087304842ada4212f2e524356 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:27:12.552407 8584 tablet_bootstrap.cc:492] T a5d47492f64f49d584246fc5007281aa P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:12.552410 8583 raft_consensus.cc:348] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:12.552623 8583 raft_consensus.cc:374] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:12.552721 8583 raft_consensus.cc:729] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:12.552942 8583 consensus_queue.cc:260] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:12.553617 8583 ts_tablet_manager.cc:1398] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:12.553908 8584 tablet_bootstrap.cc:654] T a5d47492f64f49d584246fc5007281aa P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:12.554687 8593 tablet.cc:1722] T a31bfb0283164a4da34b6af1bc5deb90 P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:12.559485 8457 raft_consensus.cc:482] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:12.559646 8457 raft_consensus.cc:504] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:12.560281 8457 leader_election.cc:288] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:12.560942 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61e18d2b13644f6a99225c106c594914" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:12.561213 7349 raft_consensus.cc:2454] T 61e18d2b13644f6a99225c106c594914 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:12.562034 7150 leader_election.cc:302] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:12.562136 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61e18d2b13644f6a99225c106c594914" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:12.562284 8457 raft_consensus.cc:2781] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:12.562402 8457 raft_consensus.cc:482] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:12.562496 8457 raft_consensus.cc:3037] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:27:12.562726 7149 leader_election.cc:341] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:12.563380 8593 tablet.cc:1620] T a31bfb0283164a4da34b6af1bc5deb90 P 73a1f2ca1f544de9b0e3b37015f85328: MemRowSet was empty: no flush needed.
I20240502 21:27:12.563727 8584 tablet_bootstrap.cc:492] T a5d47492f64f49d584246fc5007281aa P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:12.563930 8584 ts_tablet_manager.cc:1367] T a5d47492f64f49d584246fc5007281aa P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20240502 21:27:12.564565 8584 raft_consensus.cc:348] T a5d47492f64f49d584246fc5007281aa P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:12.564757 8584 raft_consensus.cc:374] T a5d47492f64f49d584246fc5007281aa P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:12.564853 8584 raft_consensus.cc:729] T a5d47492f64f49d584246fc5007281aa P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:12.565099 8584 consensus_queue.cc:260] T a5d47492f64f49d584246fc5007281aa P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:12.565069 8457 raft_consensus.cc:504] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:12.565721 8584 ts_tablet_manager.cc:1398] T a5d47492f64f49d584246fc5007281aa P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:12.565729 8457 leader_election.cc:288] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:12.566115 8584 tablet_bootstrap.cc:492] T 61e18d2b13644f6a99225c106c594914 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:12.566114 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61e18d2b13644f6a99225c106c594914" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:12.566375 7349 raft_consensus.cc:3037] T 61e18d2b13644f6a99225c106c594914 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:12.566361 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61e18d2b13644f6a99225c106c594914" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
W20240502 21:27:12.567340 7149 leader_election.cc:341] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:12.567797 8584 tablet_bootstrap.cc:654] T 61e18d2b13644f6a99225c106c594914 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:12.569214 7349 raft_consensus.cc:2454] T 61e18d2b13644f6a99225c106c594914 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:12.570055 7150 leader_election.cc:302] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, d0456c5e55a542128688904e003d403c; no voters: 0ddd486109f5434cace13d60d633ac04
I20240502 21:27:12.570159 8584 tablet_bootstrap.cc:492] T 61e18d2b13644f6a99225c106c594914 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:12.570284 8584 ts_tablet_manager.cc:1367] T 61e18d2b13644f6a99225c106c594914 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:27:12.570552 8457 raft_consensus.cc:2781] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:12.570775 8457 raft_consensus.cc:686] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:27:12.571075 8457 consensus_queue.cc:237] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:12.571156 8584 raft_consensus.cc:348] T 61e18d2b13644f6a99225c106c594914 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:12.571324 8584 raft_consensus.cc:374] T 61e18d2b13644f6a99225c106c594914 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:12.571413 8584 raft_consensus.cc:729] T 61e18d2b13644f6a99225c106c594914 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:12.571787 8584 consensus_queue.cc:260] T 61e18d2b13644f6a99225c106c594914 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:12.572342 8584 ts_tablet_manager.cc:1398] T 61e18d2b13644f6a99225c106c594914 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:12.576610 4294 catalog_manager.cc:5484] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69). New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:12.617177 8557 raft_consensus.cc:482] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:12.617363 8557 raft_consensus.cc:504] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:12.617992 8557 leader_election.cc:288] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:12.618396 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d47492f64f49d584246fc5007281aa" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:12.618413 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d47492f64f49d584246fc5007281aa" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:12.618593 7618 raft_consensus.cc:2454] T a5d47492f64f49d584246fc5007281aa P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:12.618631 7484 raft_consensus.cc:2454] T a5d47492f64f49d584246fc5007281aa P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:12.618984 7149 leader_election.cc:302] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:12.619246 8557 raft_consensus.cc:2781] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:12.619364 8557 raft_consensus.cc:482] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:12.619454 8557 raft_consensus.cc:3037] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:12.622049 8557 raft_consensus.cc:504] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:12.622584 8557 leader_election.cc:288] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:12.622993 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d47492f64f49d584246fc5007281aa" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:12.623075 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d47492f64f49d584246fc5007281aa" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:12.623175 7618 raft_consensus.cc:3037] T a5d47492f64f49d584246fc5007281aa P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:12.623291 7484 raft_consensus.cc:3037] T a5d47492f64f49d584246fc5007281aa P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:12.625033 7618 raft_consensus.cc:2454] T a5d47492f64f49d584246fc5007281aa P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:12.625151 7484 raft_consensus.cc:2454] T a5d47492f64f49d584246fc5007281aa P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:12.625509 7149 leader_election.cc:302] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:12.625806 8557 raft_consensus.cc:2781] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:12.625952 8557 raft_consensus.cc:686] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:27:12.626236 8557 consensus_queue.cc:237] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:12.629379 4300 catalog_manager.cc:5484] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69). New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:12.651423 4300 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:12.662977 8592 tablet.cc:1722] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:12.663548 8592 tablet.cc:1620] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:12.664628 8593 tablet.cc:1722] T a31bfb0283164a4da34b6af1bc5deb90 P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:12.665233 8593 tablet.cc:1620] T a31bfb0283164a4da34b6af1bc5deb90 P 73a1f2ca1f544de9b0e3b37015f85328: MemRowSet was empty: no flush needed.
I20240502 21:27:12.666893 8594 tablet.cc:1722] T a31bfb0283164a4da34b6af1bc5deb90 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:12.668946 8594 tablet.cc:1620] T a31bfb0283164a4da34b6af1bc5deb90 P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:12.677399 4300 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=bcd2a277a73646e598a0f6490ce410c1] alter complete (version 2)
I20240502 21:27:12.781384 8535 raft_consensus.cc:482] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:12.781358 4292 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:12.781540 8535 raft_consensus.cc:504] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:12.782433 8535 leader_election.cc:288] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:12.783687 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3394f5f087304842ada4212f2e524356" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:12.783987 7349 raft_consensus.cc:2454] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:27:12.784567 4641 leader_election.cc:302] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:12.785365 8535 raft_consensus.cc:2781] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:12.785509 8535 raft_consensus.cc:482] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:12.785610 8535 raft_consensus.cc:3037] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:12.791486 8592 tablet.cc:1722] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:12.795899 8535 raft_consensus.cc:504] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:12.800415 8535 leader_election.cc:288] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Requested vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:12.800755 8594 tablet.cc:1722] T a31bfb0283164a4da34b6af1bc5deb90 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:12.801167 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3394f5f087304842ada4212f2e524356" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:12.801394 7349 raft_consensus.cc:3037] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:12.804374 8593 tablet.cc:1722] T a31bfb0283164a4da34b6af1bc5deb90 P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:12.811550 7349 raft_consensus.cc:2454] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:27:12.812054 4641 leader_election.cc:302] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:12.812434 8535 raft_consensus.cc:2781] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:12.812578 8535 raft_consensus.cc:686] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Becoming Leader. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Running, Role: LEADER
I20240502 21:27:12.812853 8535 consensus_queue.cc:237] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:12.816651 4292 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=bcd2a277a73646e598a0f6490ce410c1] alter complete (version 3)
I20240502 21:27:12.819068 4292 catalog_manager.cc:5484] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 398c3c8f3195492c900439ad5e1931bb (127.29.82.67). New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:12.840032 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3394f5f087304842ada4212f2e524356" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:12.840130 7617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3394f5f087304842ada4212f2e524356" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:12.840315 7618 raft_consensus.cc:2454] T 3394f5f087304842ada4212f2e524356 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:27:12.887871 7942 raft_consensus.cc:482] T 61e18d2b13644f6a99225c106c594914 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:12.888114 7942 raft_consensus.cc:504] T 61e18d2b13644f6a99225c106c594914 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:12.888816 7942 leader_election.cc:288] T 61e18d2b13644f6a99225c106c594914 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:12.889225 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61e18d2b13644f6a99225c106c594914" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:12.889264 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61e18d2b13644f6a99225c106c594914" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:12.889503 7349 raft_consensus.cc:2379] T 61e18d2b13644f6a99225c106c594914 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ddd486109f5434cace13d60d633ac04 in current term 1: Already voted for candidate 39981a8dfe94462b9a73ba489af4c714 in this term.
I20240502 21:27:12.889967 7551 leader_election.cc:302] T 61e18d2b13644f6a99225c106c594914 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04; no voters: 39981a8dfe94462b9a73ba489af4c714, d0456c5e55a542128688904e003d403c
I20240502 21:27:12.890275 7942 raft_consensus.cc:3037] T 61e18d2b13644f6a99225c106c594914 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:12.892699 7942 raft_consensus.cc:2726] T 61e18d2b13644f6a99225c106c594914 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:27:12.919726 4292 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:12.919971 4292 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:12.923339 4292 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet a31bfb0283164a4da34b6af1bc5deb90
I20240502 21:27:12.924302 7193 tablet_service.cc:1514] Processing DeleteTablet for tablet a31bfb0283164a4da34b6af1bc5deb90 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:12 UTC) from {username='slave'} at 127.0.0.1:44652
I20240502 21:27:12.924409 4818 tablet_service.cc:1514] Processing DeleteTablet for tablet a31bfb0283164a4da34b6af1bc5deb90 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:12 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:27:12.924952 8606 tablet_replica.cc:330] T a31bfb0283164a4da34b6af1bc5deb90 P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:27:12.925225 8606 raft_consensus.cc:2229] T a31bfb0283164a4da34b6af1bc5deb90 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:12.925240 7328 tablet_service.cc:1514] Processing DeleteTablet for tablet a31bfb0283164a4da34b6af1bc5deb90 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:12 UTC) from {username='slave'} at 127.0.0.1:47078
I20240502 21:27:12.925493 8606 raft_consensus.cc:2258] T a31bfb0283164a4da34b6af1bc5deb90 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:12.926065 8608 tablet_replica.cc:330] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:27:12.926332 8608 raft_consensus.cc:2229] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:12.926589 8606 ts_tablet_manager.cc:1875] T a31bfb0283164a4da34b6af1bc5deb90 P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:12.926934 8608 raft_consensus.cc:2258] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:12.927923 8608 ts_tablet_manager.cc:1875] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:12.933538 8606 ts_tablet_manager.cc:1888] T a31bfb0283164a4da34b6af1bc5deb90 P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:12.933670 8606 log.cc:1196] T a31bfb0283164a4da34b6af1bc5deb90 P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/a31bfb0283164a4da34b6af1bc5deb90
I20240502 21:27:12.934144 8606 ts_tablet_manager.cc:1909] T a31bfb0283164a4da34b6af1bc5deb90 P 39981a8dfe94462b9a73ba489af4c714: Deleting consensus metadata
I20240502 21:27:12.934408 8608 ts_tablet_manager.cc:1888] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:12.934589 8608 log.cc:1196] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/a31bfb0283164a4da34b6af1bc5deb90
I20240502 21:27:12.935079 8608 ts_tablet_manager.cc:1909] T a31bfb0283164a4da34b6af1bc5deb90 P d0456c5e55a542128688904e003d403c: Deleting consensus metadata
I20240502 21:27:12.935449 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet a31bfb0283164a4da34b6af1bc5deb90 (table rebalancer_dynamic_table_new_name [id=bcd2a277a73646e598a0f6490ce410c1]) successfully deleted
I20240502 21:27:12.936298 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet a31bfb0283164a4da34b6af1bc5deb90 (table rebalancer_dynamic_table_new_name [id=bcd2a277a73646e598a0f6490ce410c1]) successfully deleted
I20240502 21:27:12.936601 8607 tablet_replica.cc:330] T a31bfb0283164a4da34b6af1bc5deb90 P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:27:12.936857 8607 raft_consensus.cc:2229] T a31bfb0283164a4da34b6af1bc5deb90 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:12.937073 8607 raft_consensus.cc:2258] T a31bfb0283164a4da34b6af1bc5deb90 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:12.937842 8607 ts_tablet_manager.cc:1875] T a31bfb0283164a4da34b6af1bc5deb90 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:12.943178 8607 ts_tablet_manager.cc:1888] T a31bfb0283164a4da34b6af1bc5deb90 P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:12.943289 8607 log.cc:1196] T a31bfb0283164a4da34b6af1bc5deb90 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/a31bfb0283164a4da34b6af1bc5deb90
I20240502 21:27:12.943702 8607 ts_tablet_manager.cc:1909] T a31bfb0283164a4da34b6af1bc5deb90 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:27:12.944705 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet a31bfb0283164a4da34b6af1bc5deb90 (table rebalancer_dynamic_table_new_name [id=bcd2a277a73646e598a0f6490ce410c1]) successfully deleted
I20240502 21:27:13.026033 4292 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:13.033747 4555 tablet_service.cc:1467] Processing CreateTablet for tablet fb6e7b6555824ab39f1a523c946fb003 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=721a45197a6d4a10b884e254f4d5fa3f]), partition=
I20240502 21:27:13.034129 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6e7b6555824ab39f1a523c946fb003. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:13.039386 8609 tablet_bootstrap.cc:492] T fb6e7b6555824ab39f1a523c946fb003 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:13.039784 7193 tablet_service.cc:1467] Processing CreateTablet for tablet fb6e7b6555824ab39f1a523c946fb003 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=721a45197a6d4a10b884e254f4d5fa3f]), partition=
I20240502 21:27:13.040246 7193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6e7b6555824ab39f1a523c946fb003. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:13.040279 8457 consensus_queue.cc:1035] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:13.041255 8609 tablet_bootstrap.cc:654] T fb6e7b6555824ab39f1a523c946fb003 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:13.044209 4818 tablet_service.cc:1467] Processing CreateTablet for tablet fb6e7b6555824ab39f1a523c946fb003 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=721a45197a6d4a10b884e254f4d5fa3f]), partition=
I20240502 21:27:13.044553 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6e7b6555824ab39f1a523c946fb003. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:13.048089 8612 tablet_bootstrap.cc:492] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:13.050349 8612 tablet_bootstrap.cc:654] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:13.052986 8609 tablet_bootstrap.cc:492] T fb6e7b6555824ab39f1a523c946fb003 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:13.053164 8609 ts_tablet_manager.cc:1367] T fb6e7b6555824ab39f1a523c946fb003 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20240502 21:27:13.054785 8609 raft_consensus.cc:348] T fb6e7b6555824ab39f1a523c946fb003 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:13.055161 8609 raft_consensus.cc:374] T fb6e7b6555824ab39f1a523c946fb003 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:13.056299 8613 tablet_bootstrap.cc:492] T fb6e7b6555824ab39f1a523c946fb003 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:13.057228 8457 consensus_queue.cc:1035] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:13.060801 8613 tablet_bootstrap.cc:654] T fb6e7b6555824ab39f1a523c946fb003 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:13.055349 8609 raft_consensus.cc:729] T fb6e7b6555824ab39f1a523c946fb003 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:13.063689 8613 tablet_bootstrap.cc:492] T fb6e7b6555824ab39f1a523c946fb003 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:13.063841 8613 ts_tablet_manager.cc:1367] T fb6e7b6555824ab39f1a523c946fb003 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
I20240502 21:27:13.063460 8609 consensus_queue.cc:260] T fb6e7b6555824ab39f1a523c946fb003 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:13.064620 8613 raft_consensus.cc:348] T fb6e7b6555824ab39f1a523c946fb003 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:13.064853 8613 raft_consensus.cc:374] T fb6e7b6555824ab39f1a523c946fb003 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:13.064966 8613 raft_consensus.cc:729] T fb6e7b6555824ab39f1a523c946fb003 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:13.065244 8613 consensus_queue.cc:260] T fb6e7b6555824ab39f1a523c946fb003 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:13.065347 8609 ts_tablet_manager.cc:1398] T fb6e7b6555824ab39f1a523c946fb003 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.012s user 0.001s sys 0.002s
I20240502 21:27:13.065925 8613 ts_tablet_manager.cc:1398] T fb6e7b6555824ab39f1a523c946fb003 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:13.070812 8612 tablet_bootstrap.cc:492] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:13.071013 8612 ts_tablet_manager.cc:1367] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.000s
I20240502 21:27:13.072103 8612 raft_consensus.cc:348] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:13.072290 8612 raft_consensus.cc:374] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:13.072396 8612 raft_consensus.cc:729] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:13.072620 8612 consensus_queue.cc:260] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:13.073208 8612 ts_tablet_manager.cc:1398] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:13.095383 8457 raft_consensus.cc:482] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:13.095640 8457 raft_consensus.cc:504] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:13.096417 8457 leader_election.cc:288] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:13.097030 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6e7b6555824ab39f1a523c946fb003" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:27:13.097275 4575 raft_consensus.cc:2454] T fb6e7b6555824ab39f1a523c946fb003 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:13.097211 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6e7b6555824ab39f1a523c946fb003" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:27:13.097491 4839 raft_consensus.cc:2454] T fb6e7b6555824ab39f1a523c946fb003 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:13.098019 7151 leader_election.cc:302] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:27:13.098325 8457 raft_consensus.cc:2781] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:13.098434 8457 raft_consensus.cc:482] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:13.098543 8457 raft_consensus.cc:3037] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:13.100629 8457 raft_consensus.cc:504] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:13.101239 8457 leader_election.cc:288] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:13.101696 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6e7b6555824ab39f1a523c946fb003" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:27:13.101938 4839 raft_consensus.cc:3037] T fb6e7b6555824ab39f1a523c946fb003 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:13.101918 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6e7b6555824ab39f1a523c946fb003" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:27:13.102118 4575 raft_consensus.cc:3037] T fb6e7b6555824ab39f1a523c946fb003 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:13.104343 4839 raft_consensus.cc:2454] T fb6e7b6555824ab39f1a523c946fb003 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:13.104387 4575 raft_consensus.cc:2454] T fb6e7b6555824ab39f1a523c946fb003 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:13.104861 7148 leader_election.cc:302] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:13.105201 8457 raft_consensus.cc:2781] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:13.105381 8457 raft_consensus.cc:686] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:27:13.105659 8457 consensus_queue.cc:237] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:13.108913 4298 catalog_manager.cc:5484] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69). New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:13.146875 8457 consensus_queue.cc:1035] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:13.150727 8557 consensus_queue.cc:1035] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:13.229055 4298 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:13.235872 4839 raft_consensus.cc:1261] T fb6e7b6555824ab39f1a523c946fb003 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:13.236429 4575 raft_consensus.cc:1261] T fb6e7b6555824ab39f1a523c946fb003 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:13.237234 8457 consensus_queue.cc:1035] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:13.237777 8457 consensus_queue.cc:1035] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:13.250958 4300 catalog_manager.cc:5576] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449) does not have the latest schema for tablet fb6e7b6555824ab39f1a523c946fb003 (table rebalancer_dynamic_table [id=721a45197a6d4a10b884e254f4d5fa3f]). Expected version 1 got 0
I20240502 21:27:13.251004 8594 tablet.cc:1722] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:13.251525 8594 tablet.cc:1620] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:13.252339 8625 tablet.cc:1722] T fb6e7b6555824ab39f1a523c946fb003 P 7a736a8ebb164ce0a051c0fccd76c3b8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:13.253000 8625 tablet.cc:1620] T fb6e7b6555824ab39f1a523c946fb003 P 7a736a8ebb164ce0a051c0fccd76c3b8: MemRowSet was empty: no flush needed.
I20240502 21:27:13.255055 8593 tablet.cc:1722] T fb6e7b6555824ab39f1a523c946fb003 P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:13.255609 8593 tablet.cc:1620] T fb6e7b6555824ab39f1a523c946fb003 P 73a1f2ca1f544de9b0e3b37015f85328: MemRowSet was empty: no flush needed.
I20240502 21:27:13.256992 4298 catalog_manager.cc:5576] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781) does not have the latest schema for tablet fb6e7b6555824ab39f1a523c946fb003 (table rebalancer_dynamic_table [id=721a45197a6d4a10b884e254f4d5fa3f]). Expected version 1 got 0
I20240502 21:27:13.264369 4300 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=721a45197a6d4a10b884e254f4d5fa3f] alter complete (version 1)
I20240502 21:27:13.324622 8535 consensus_queue.cc:1035] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:13.340620 7618 raft_consensus.cc:3037] T 3394f5f087304842ada4212f2e524356 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:13.344084 8628 consensus_queue.cc:1035] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:13.371096 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:13.378583 8594 tablet.cc:1722] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:13.379187 8594 tablet.cc:1620] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:13.380405 8625 tablet.cc:1722] T fb6e7b6555824ab39f1a523c946fb003 P 7a736a8ebb164ce0a051c0fccd76c3b8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:13.380903 8625 tablet.cc:1620] T fb6e7b6555824ab39f1a523c946fb003 P 7a736a8ebb164ce0a051c0fccd76c3b8: MemRowSet was empty: no flush needed.
I20240502 21:27:13.381908 8593 tablet.cc:1722] T fb6e7b6555824ab39f1a523c946fb003 P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:13.382416 8593 tablet.cc:1620] T fb6e7b6555824ab39f1a523c946fb003 P 73a1f2ca1f544de9b0e3b37015f85328: MemRowSet was empty: no flush needed.
I20240502 21:27:13.387396 4294 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:13.390561 4298 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=721a45197a6d4a10b884e254f4d5fa3f] alter complete (version 2)
I20240502 21:27:13.398375 7328 tablet_service.cc:1467] Processing CreateTablet for tablet 269eb0e719804925961e4bcd17c18596 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=9606abae682a4594ada645db2e56a7d4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:13.398492 7597 tablet_service.cc:1467] Processing CreateTablet for tablet 269eb0e719804925961e4bcd17c18596 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=9606abae682a4594ada645db2e56a7d4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:13.398967 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 269eb0e719804925961e4bcd17c18596. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:13.399001 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 269eb0e719804925961e4bcd17c18596. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:13.398929 7464 tablet_service.cc:1467] Processing CreateTablet for tablet 269eb0e719804925961e4bcd17c18596 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=9606abae682a4594ada645db2e56a7d4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:13.399369 7464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 269eb0e719804925961e4bcd17c18596. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:13.400470 7595 tablet_service.cc:1467] Processing CreateTablet for tablet 8991236419e642bfa438b9e45a1b8235 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=9606abae682a4594ada645db2e56a7d4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:13.400970 7595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8991236419e642bfa438b9e45a1b8235. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:13.404433 7329 tablet_service.cc:1467] Processing CreateTablet for tablet 8991236419e642bfa438b9e45a1b8235 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=9606abae682a4594ada645db2e56a7d4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:13.404870 7329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8991236419e642bfa438b9e45a1b8235. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:13.409307 7596 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc82d5106804490993007f9f75943aa (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=9606abae682a4594ada645db2e56a7d4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:13.409720 7596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc82d5106804490993007f9f75943aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:13.410897 7462 tablet_service.cc:1467] Processing CreateTablet for tablet 8991236419e642bfa438b9e45a1b8235 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=9606abae682a4594ada645db2e56a7d4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:13.411316 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8991236419e642bfa438b9e45a1b8235. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:13.413318 7463 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc82d5106804490993007f9f75943aa (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=9606abae682a4594ada645db2e56a7d4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:13.413688 7463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc82d5106804490993007f9f75943aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:13.415855 8633 tablet_bootstrap.cc:492] T 269eb0e719804925961e4bcd17c18596 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:13.416815 7193 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc82d5106804490993007f9f75943aa (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=9606abae682a4594ada645db2e56a7d4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:13.417266 7193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc82d5106804490993007f9f75943aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:13.419996 8633 tablet_bootstrap.cc:654] T 269eb0e719804925961e4bcd17c18596 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:13.420039 8634 tablet_bootstrap.cc:492] T 8991236419e642bfa438b9e45a1b8235 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:13.422017 8634 tablet_bootstrap.cc:654] T 8991236419e642bfa438b9e45a1b8235 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:13.423352 8632 tablet_bootstrap.cc:492] T 269eb0e719804925961e4bcd17c18596 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:13.425014 8632 tablet_bootstrap.cc:654] T 269eb0e719804925961e4bcd17c18596 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:13.426853 8634 tablet_bootstrap.cc:492] T 8991236419e642bfa438b9e45a1b8235 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:13.427055 8634 ts_tablet_manager.cc:1367] T 8991236419e642bfa438b9e45a1b8235 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 21:27:13.427893 8634 raft_consensus.cc:348] T 8991236419e642bfa438b9e45a1b8235 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:13.428259 8634 raft_consensus.cc:374] T 8991236419e642bfa438b9e45a1b8235 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:13.428413 8634 raft_consensus.cc:729] T 8991236419e642bfa438b9e45a1b8235 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:13.428781 8633 tablet_bootstrap.cc:492] T 269eb0e719804925961e4bcd17c18596 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:13.428768 8634 consensus_queue.cc:260] T 8991236419e642bfa438b9e45a1b8235 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:13.429157 8633 ts_tablet_manager.cc:1367] T 269eb0e719804925961e4bcd17c18596 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20240502 21:27:13.429982 8633 raft_consensus.cc:348] T 269eb0e719804925961e4bcd17c18596 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:13.430240 8633 raft_consensus.cc:374] T 269eb0e719804925961e4bcd17c18596 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:13.430406 8633 raft_consensus.cc:729] T 269eb0e719804925961e4bcd17c18596 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:13.430552 8634 ts_tablet_manager.cc:1398] T 8991236419e642bfa438b9e45a1b8235 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:13.430846 8634 tablet_bootstrap.cc:492] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:13.431142 8633 consensus_queue.cc:260] T 269eb0e719804925961e4bcd17c18596 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:13.432426 8634 tablet_bootstrap.cc:654] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:13.434514 8612 tablet_bootstrap.cc:492] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:13.435365 8633 ts_tablet_manager.cc:1398] T 269eb0e719804925961e4bcd17c18596 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 21:27:13.436784 8612 tablet_bootstrap.cc:654] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:13.439394 8633 tablet_bootstrap.cc:492] T 8991236419e642bfa438b9e45a1b8235 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:13.439445 8612 tablet_bootstrap.cc:492] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:13.439787 8612 ts_tablet_manager.cc:1367] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240502 21:27:13.440776 8612 raft_consensus.cc:348] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:13.441011 8633 tablet_bootstrap.cc:654] T 8991236419e642bfa438b9e45a1b8235 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:13.441190 8612 raft_consensus.cc:374] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:13.441357 8612 raft_consensus.cc:729] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:13.441756 8612 consensus_queue.cc:260] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:13.443328 8632 tablet_bootstrap.cc:492] T 269eb0e719804925961e4bcd17c18596 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:13.443513 8632 ts_tablet_manager.cc:1367] T 269eb0e719804925961e4bcd17c18596 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.003s
I20240502 21:27:13.444597 8632 raft_consensus.cc:348] T 269eb0e719804925961e4bcd17c18596 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:13.444870 8632 raft_consensus.cc:374] T 269eb0e719804925961e4bcd17c18596 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:13.444969 8632 raft_consensus.cc:729] T 269eb0e719804925961e4bcd17c18596 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:13.445255 8632 consensus_queue.cc:260] T 269eb0e719804925961e4bcd17c18596 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:13.447513 8612 ts_tablet_manager.cc:1398] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240502 21:27:13.449097 8633 tablet_bootstrap.cc:492] T 8991236419e642bfa438b9e45a1b8235 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:13.449252 8633 ts_tablet_manager.cc:1367] T 8991236419e642bfa438b9e45a1b8235 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240502 21:27:13.450074 8633 raft_consensus.cc:348] T 8991236419e642bfa438b9e45a1b8235 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:13.450294 8633 raft_consensus.cc:374] T 8991236419e642bfa438b9e45a1b8235 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:13.450372 8633 raft_consensus.cc:729] T 8991236419e642bfa438b9e45a1b8235 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:13.451943 8632 ts_tablet_manager.cc:1398] T 269eb0e719804925961e4bcd17c18596 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 21:27:13.452252 8632 tablet_bootstrap.cc:492] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:13.454377 8632 tablet_bootstrap.cc:654] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:13.454919 8634 tablet_bootstrap.cc:492] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:13.455142 8634 ts_tablet_manager.cc:1367] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.008s
I20240502 21:27:13.456049 8634 raft_consensus.cc:348] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:13.456262 8634 raft_consensus.cc:374] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:13.456343 8634 raft_consensus.cc:729] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:13.456585 8634 consensus_queue.cc:260] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:13.457306 8634 ts_tablet_manager.cc:1398] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:13.457664 8634 tablet_bootstrap.cc:492] T 4cc82d5106804490993007f9f75943aa P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:13.457116 8633 consensus_queue.cc:260] T 8991236419e642bfa438b9e45a1b8235 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:13.459435 8634 tablet_bootstrap.cc:654] T 4cc82d5106804490993007f9f75943aa P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:13.463359 8633 ts_tablet_manager.cc:1398] T 8991236419e642bfa438b9e45a1b8235 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20240502 21:27:13.463400 8526 raft_consensus.cc:482] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:13.463620 8526 raft_consensus.cc:504] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:13.464444 8526 leader_election.cc:288] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:13.465384 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "269eb0e719804925961e4bcd17c18596" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:13.465641 7618 raft_consensus.cc:2454] T 269eb0e719804925961e4bcd17c18596 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:13.465703 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "269eb0e719804925961e4bcd17c18596" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:13.465888 7350 raft_consensus.cc:2454] T 269eb0e719804925961e4bcd17c18596 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:13.466138 7418 leader_election.cc:302] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168; no voters:
I20240502 21:27:13.466540 8526 raft_consensus.cc:2781] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:13.466641 8526 raft_consensus.cc:482] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:13.466758 8526 raft_consensus.cc:3037] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:13.467092 8634 tablet_bootstrap.cc:492] T 4cc82d5106804490993007f9f75943aa P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:13.467308 8634 ts_tablet_manager.cc:1367] T 4cc82d5106804490993007f9f75943aa P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20240502 21:27:13.467787 8632 tablet_bootstrap.cc:492] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:13.467983 8632 ts_tablet_manager.cc:1367] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.002s
I20240502 21:27:13.468149 8634 raft_consensus.cc:348] T 4cc82d5106804490993007f9f75943aa P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:13.468377 8634 raft_consensus.cc:374] T 4cc82d5106804490993007f9f75943aa P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:13.468478 8634 raft_consensus.cc:729] T 4cc82d5106804490993007f9f75943aa P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:13.468710 8634 consensus_queue.cc:260] T 4cc82d5106804490993007f9f75943aa P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:13.469149 8632 raft_consensus.cc:348] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:13.469419 8632 raft_consensus.cc:374] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:13.469584 8632 raft_consensus.cc:729] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:13.469942 8632 consensus_queue.cc:260] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:13.470129 8526 raft_consensus.cc:504] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:13.471814 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "269eb0e719804925961e4bcd17c18596" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:13.471984 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "269eb0e719804925961e4bcd17c18596" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:13.472026 7350 raft_consensus.cc:3037] T 269eb0e719804925961e4bcd17c18596 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:13.472178 7618 raft_consensus.cc:3037] T 269eb0e719804925961e4bcd17c18596 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:13.472259 8526 leader_election.cc:288] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:13.474189 8632 ts_tablet_manager.cc:1398] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 21:27:13.474767 7350 raft_consensus.cc:2454] T 269eb0e719804925961e4bcd17c18596 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:13.474938 7618 raft_consensus.cc:2454] T 269eb0e719804925961e4bcd17c18596 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:13.475315 7418 leader_election.cc:302] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168; no voters:
I20240502 21:27:13.475512 8632 tablet_bootstrap.cc:492] T 4cc82d5106804490993007f9f75943aa P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:13.475682 8526 raft_consensus.cc:2781] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:13.475875 8526 raft_consensus.cc:686] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Becoming Leader. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Running, Role: LEADER
I20240502 21:27:13.476154 8526 consensus_queue.cc:237] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:13.482300 8634 ts_tablet_manager.cc:1398] T 4cc82d5106804490993007f9f75943aa P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.015s user 0.005s sys 0.003s
I20240502 21:27:13.484552 8632 tablet_bootstrap.cc:654] T 4cc82d5106804490993007f9f75943aa P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:13.494560 8632 tablet_bootstrap.cc:492] T 4cc82d5106804490993007f9f75943aa P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:13.495599 8632 ts_tablet_manager.cc:1367] T 4cc82d5106804490993007f9f75943aa P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.001s
I20240502 21:27:13.495844 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:13.496810 8632 raft_consensus.cc:348] T 4cc82d5106804490993007f9f75943aa P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:13.497239 8632 raft_consensus.cc:374] T 4cc82d5106804490993007f9f75943aa P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:13.497404 8632 raft_consensus.cc:729] T 4cc82d5106804490993007f9f75943aa P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:13.498373 8632 consensus_queue.cc:260] T 4cc82d5106804490993007f9f75943aa P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:13.500530 8557 raft_consensus.cc:482] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:13.500706 8557 raft_consensus.cc:504] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:13.501435 8557 leader_election.cc:288] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:13.502244 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc82d5106804490993007f9f75943aa" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:13.502497 7618 raft_consensus.cc:2454] T 4cc82d5106804490993007f9f75943aa P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:13.503232 7149 leader_election.cc:302] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:13.503693 8632 ts_tablet_manager.cc:1398] T 4cc82d5106804490993007f9f75943aa P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 21:27:13.503926 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc82d5106804490993007f9f75943aa" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:13.504148 7484 raft_consensus.cc:2454] T 4cc82d5106804490993007f9f75943aa P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:13.505282 8557 raft_consensus.cc:2781] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:13.505409 8557 raft_consensus.cc:482] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:13.505501 8557 raft_consensus.cc:3037] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:13.508630 8557 raft_consensus.cc:504] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:13.509662 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc82d5106804490993007f9f75943aa" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:13.509711 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc82d5106804490993007f9f75943aa" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:13.509878 7484 raft_consensus.cc:3037] T 4cc82d5106804490993007f9f75943aa P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:13.509896 7618 raft_consensus.cc:3037] T 4cc82d5106804490993007f9f75943aa P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:13.511142 8593 tablet.cc:1722] T fb6e7b6555824ab39f1a523c946fb003 P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:13.511724 8594 tablet.cc:1722] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:13.513356 7484 raft_consensus.cc:2454] T 4cc82d5106804490993007f9f75943aa P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:13.516041 7148 leader_election.cc:302] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:13.517987 7618 raft_consensus.cc:2454] T 4cc82d5106804490993007f9f75943aa P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:13.518580 8557 leader_election.cc:288] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:13.518602 8651 raft_consensus.cc:2781] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:13.519353 8651 raft_consensus.cc:686] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:27:13.519758 8651 consensus_queue.cc:237] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:13.522914 4300 catalog_manager.cc:5484] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a72286397c4edc9a63eca3412be168 (127.29.82.71). New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:27:13.526554 8625 tablet.cc:1722] T fb6e7b6555824ab39f1a523c946fb003 P 7a736a8ebb164ce0a051c0fccd76c3b8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:13.529167 4294 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=721a45197a6d4a10b884e254f4d5fa3f] alter complete (version 3)
I20240502 21:27:13.548758 4291 catalog_manager.cc:5484] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69). New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:13.645975 8626 raft_consensus.cc:482] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:13.646171 8626 raft_consensus.cc:504] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:13.646962 8626 leader_election.cc:288] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:13.647325 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8991236419e642bfa438b9e45a1b8235" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:13.647532 7350 raft_consensus.cc:2454] T 8991236419e642bfa438b9e45a1b8235 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:13.647503 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8991236419e642bfa438b9e45a1b8235" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:13.647830 7484 raft_consensus.cc:2454] T 8991236419e642bfa438b9e45a1b8235 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:13.648010 7551 leader_election.cc:302] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:13.648344 8626 raft_consensus.cc:2781] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:13.648469 8626 raft_consensus.cc:482] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:13.648546 8626 raft_consensus.cc:3037] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:13.650427 8626 raft_consensus.cc:504] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:13.650974 8626 leader_election.cc:288] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:13.651414 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8991236419e642bfa438b9e45a1b8235" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:13.651501 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8991236419e642bfa438b9e45a1b8235" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:13.651672 7350 raft_consensus.cc:3037] T 8991236419e642bfa438b9e45a1b8235 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:13.651795 7484 raft_consensus.cc:3037] T 8991236419e642bfa438b9e45a1b8235 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:13.651886 4291 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:13.652122 4291 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:13.654394 7484 raft_consensus.cc:2454] T 8991236419e642bfa438b9e45a1b8235 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:13.654541 7350 raft_consensus.cc:2454] T 8991236419e642bfa438b9e45a1b8235 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:13.654922 7551 leader_election.cc:302] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:13.655197 8626 raft_consensus.cc:2781] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:13.655369 8626 raft_consensus.cc:686] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Becoming Leader. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Running, Role: LEADER
I20240502 21:27:13.655678 8626 consensus_queue.cc:237] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:13.659341 4291 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet fb6e7b6555824ab39f1a523c946fb003
I20240502 21:27:13.660302 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet fb6e7b6555824ab39f1a523c946fb003 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:13 UTC) from {username='slave'} at 127.0.0.1:60018
I20240502 21:27:13.660466 4818 tablet_service.cc:1514] Processing DeleteTablet for tablet fb6e7b6555824ab39f1a523c946fb003 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:13 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:27:13.661140 7193 tablet_service.cc:1514] Processing DeleteTablet for tablet fb6e7b6555824ab39f1a523c946fb003 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:13 UTC) from {username='slave'} at 127.0.0.1:44652
I20240502 21:27:13.660874 4300 catalog_manager.cc:5484] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ddd486109f5434cace13d60d633ac04 (127.29.82.72). New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:13.662737 8657 tablet_replica.cc:330] T fb6e7b6555824ab39f1a523c946fb003 P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:27:13.663029 8656 tablet_replica.cc:330] T fb6e7b6555824ab39f1a523c946fb003 P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:27:13.663053 8658 tablet_replica.cc:330] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:27:13.663286 8656 raft_consensus.cc:2229] T fb6e7b6555824ab39f1a523c946fb003 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:13.663275 8658 raft_consensus.cc:2229] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:13.663533 8656 raft_consensus.cc:2258] T fb6e7b6555824ab39f1a523c946fb003 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:13.663666 8658 raft_consensus.cc:2258] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:13.664376 8656 ts_tablet_manager.cc:1875] T fb6e7b6555824ab39f1a523c946fb003 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:13.664697 8657 raft_consensus.cc:2229] T fb6e7b6555824ab39f1a523c946fb003 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:13.664983 8657 raft_consensus.cc:2258] T fb6e7b6555824ab39f1a523c946fb003 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:13.666290 8658 ts_tablet_manager.cc:1875] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:13.667697 8657 ts_tablet_manager.cc:1875] T fb6e7b6555824ab39f1a523c946fb003 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:13.673482 8656 ts_tablet_manager.cc:1888] T fb6e7b6555824ab39f1a523c946fb003 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:13.673620 8656 log.cc:1196] T fb6e7b6555824ab39f1a523c946fb003 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/fb6e7b6555824ab39f1a523c946fb003
I20240502 21:27:13.674134 8656 ts_tablet_manager.cc:1909] T fb6e7b6555824ab39f1a523c946fb003 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting consensus metadata
I20240502 21:27:13.674940 8658 ts_tablet_manager.cc:1888] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:13.675128 8658 log.cc:1196] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/fb6e7b6555824ab39f1a523c946fb003
I20240502 21:27:13.675664 8658 ts_tablet_manager.cc:1909] T fb6e7b6555824ab39f1a523c946fb003 P 39981a8dfe94462b9a73ba489af4c714: Deleting consensus metadata
I20240502 21:27:13.676718 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet fb6e7b6555824ab39f1a523c946fb003 (table rebalancer_dynamic_table_new_name [id=721a45197a6d4a10b884e254f4d5fa3f]) successfully deleted
I20240502 21:27:13.678902 8657 ts_tablet_manager.cc:1888] T fb6e7b6555824ab39f1a523c946fb003 P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:13.679054 8657 log.cc:1196] T fb6e7b6555824ab39f1a523c946fb003 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/fb6e7b6555824ab39f1a523c946fb003
I20240502 21:27:13.679459 8657 ts_tablet_manager.cc:1909] T fb6e7b6555824ab39f1a523c946fb003 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:27:13.680419 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet fb6e7b6555824ab39f1a523c946fb003 (table rebalancer_dynamic_table_new_name [id=721a45197a6d4a10b884e254f4d5fa3f]) successfully deleted
I20240502 21:27:13.681118 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet fb6e7b6555824ab39f1a523c946fb003 (table rebalancer_dynamic_table_new_name [id=721a45197a6d4a10b884e254f4d5fa3f]) successfully deleted
I20240502 21:27:13.762084 4300 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:13.770884 7328 tablet_service.cc:1467] Processing CreateTablet for tablet 6af430a9878a4a86b9057f94e3893510 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=04480d680768439098c8518baa3d63ee]), partition=
I20240502 21:27:13.771204 7463 tablet_service.cc:1467] Processing CreateTablet for tablet 6af430a9878a4a86b9057f94e3893510 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=04480d680768439098c8518baa3d63ee]), partition=
I20240502 21:27:13.771291 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af430a9878a4a86b9057f94e3893510. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:13.771553 7463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af430a9878a4a86b9057f94e3893510. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:13.772193 7193 tablet_service.cc:1467] Processing CreateTablet for tablet 6af430a9878a4a86b9057f94e3893510 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=04480d680768439098c8518baa3d63ee]), partition=
I20240502 21:27:13.772536 7193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af430a9878a4a86b9057f94e3893510. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:13.775992 8634 tablet_bootstrap.cc:492] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:13.777499 8634 tablet_bootstrap.cc:654] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:13.781419 8634 tablet_bootstrap.cc:492] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:13.781630 8634 ts_tablet_manager.cc:1367] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240502 21:27:13.782475 8634 raft_consensus.cc:348] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:13.782735 8634 raft_consensus.cc:374] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:13.782830 8634 raft_consensus.cc:729] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:13.783099 8634 consensus_queue.cc:260] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:13.783772 8634 ts_tablet_manager.cc:1398] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:13.785965 8526 raft_consensus.cc:482] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:13.786124 8526 raft_consensus.cc:504] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:13.786947 8526 leader_election.cc:288] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
W20240502 21:27:13.788702 7419 leader_election.cc:341] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Not found: Tablet not found: 6af430a9878a4a86b9057f94e3893510
W20240502 21:27:13.788919 7419 leader_election.cc:341] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Not found: Tablet not found: 6af430a9878a4a86b9057f94e3893510
I20240502 21:27:13.789036 7419 leader_election.cc:302] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a72286397c4edc9a63eca3412be168; no voters: 39981a8dfe94462b9a73ba489af4c714, d0456c5e55a542128688904e003d403c
I20240502 21:27:13.789971 8633 tablet_bootstrap.cc:492] T 6af430a9878a4a86b9057f94e3893510 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:13.790217 8526 raft_consensus.cc:2726] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:27:13.791695 8633 tablet_bootstrap.cc:654] T 6af430a9878a4a86b9057f94e3893510 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:13.794199 8633 tablet_bootstrap.cc:492] T 6af430a9878a4a86b9057f94e3893510 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:13.794219 8612 tablet_bootstrap.cc:492] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:13.794364 8633 ts_tablet_manager.cc:1367] T 6af430a9878a4a86b9057f94e3893510 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240502 21:27:13.795221 8633 raft_consensus.cc:348] T 6af430a9878a4a86b9057f94e3893510 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:13.795531 8633 raft_consensus.cc:374] T 6af430a9878a4a86b9057f94e3893510 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:13.795729 8633 raft_consensus.cc:729] T 6af430a9878a4a86b9057f94e3893510 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:13.795902 8612 tablet_bootstrap.cc:654] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:13.796104 8633 consensus_queue.cc:260] T 6af430a9878a4a86b9057f94e3893510 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:13.801303 8633 ts_tablet_manager.cc:1398] T 6af430a9878a4a86b9057f94e3893510 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20240502 21:27:13.801734 8612 tablet_bootstrap.cc:492] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:13.801884 8612 ts_tablet_manager.cc:1367] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20240502 21:27:13.802695 8612 raft_consensus.cc:348] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:13.802896 8612 raft_consensus.cc:374] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:13.803036 8612 raft_consensus.cc:729] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:13.803668 8612 consensus_queue.cc:260] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:13.804366 8612 ts_tablet_manager.cc:1398] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:13.823093 8649 raft_consensus.cc:482] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:13.823266 8649 raft_consensus.cc:504] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:13.824030 8649 leader_election.cc:288] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:13.824468 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af430a9878a4a86b9057f94e3893510" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:13.824496 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af430a9878a4a86b9057f94e3893510" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:13.824717 7484 raft_consensus.cc:2454] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:13.824728 7349 raft_consensus.cc:2454] T 6af430a9878a4a86b9057f94e3893510 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:13.825105 7150 leader_election.cc:302] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:13.825382 8649 raft_consensus.cc:2781] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:13.825501 8649 raft_consensus.cc:482] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:13.825582 8649 raft_consensus.cc:3037] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:13.828893 8649 raft_consensus.cc:504] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:13.829938 8649 leader_election.cc:288] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:13.830220 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af430a9878a4a86b9057f94e3893510" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:13.830423 7484 raft_consensus.cc:3037] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:13.833148 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af430a9878a4a86b9057f94e3893510" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:13.833360 7349 raft_consensus.cc:3037] T 6af430a9878a4a86b9057f94e3893510 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:13.838492 7349 raft_consensus.cc:2454] T 6af430a9878a4a86b9057f94e3893510 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:13.838973 7150 leader_election.cc:302] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:13.839038 7484 raft_consensus.cc:2454] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:13.840286 8649 raft_consensus.cc:2781] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:13.840451 8649 raft_consensus.cc:686] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:27:13.840742 8649 consensus_queue.cc:237] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:13.849942 4292 catalog_manager.cc:5484] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69). New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:27:13.873996 8526 consensus_queue.cc:1035] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:13.878904 8526 consensus_queue.cc:1035] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:13.954442 8649 consensus_queue.cc:1035] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:13.958597 8651 consensus_queue.cc:1035] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:13.965801 4300 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:13.970546 7484 raft_consensus.cc:1261] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:13.970706 7349 raft_consensus.cc:1261] T 6af430a9878a4a86b9057f94e3893510 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:13.971113 8649 consensus_queue.cc:1035] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:13.971355 8651 consensus_queue.cc:1035] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:13.975747 4300 catalog_manager.cc:5576] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719) does not have the latest schema for tablet 6af430a9878a4a86b9057f94e3893510 (table rebalancer_dynamic_table [id=04480d680768439098c8518baa3d63ee]). Expected version 1 got 0
I20240502 21:27:13.976092 8594 tablet.cc:1722] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:13.976687 8594 tablet.cc:1620] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:13.976819 8674 tablet.cc:1722] T 6af430a9878a4a86b9057f94e3893510 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:13.977355 8674 tablet.cc:1620] T 6af430a9878a4a86b9057f94e3893510 P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:13.979760 4300 catalog_manager.cc:5576] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719) does not have the latest schema for tablet 6af430a9878a4a86b9057f94e3893510 (table rebalancer_dynamic_table [id=04480d680768439098c8518baa3d63ee]). Expected version 1 got 0
I20240502 21:27:13.986810 4292 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=04480d680768439098c8518baa3d63ee] alter complete (version 1)
I20240502 21:27:13.992882 4300 catalog_manager.cc:5576] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913) does not have the latest schema for tablet 6af430a9878a4a86b9057f94e3893510 (table rebalancer_dynamic_table [id=04480d680768439098c8518baa3d63ee]). Expected version 1 got 0
I20240502 21:27:13.996127 8676 tablet.cc:1722] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:13.996771 8676 tablet.cc:1620] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:13.996840 4300 catalog_manager.cc:5576] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913) does not have the latest schema for tablet 6af430a9878a4a86b9057f94e3893510 (table rebalancer_dynamic_table [id=04480d680768439098c8518baa3d63ee]). Expected version 1 got 0
I20240502 21:27:14.054921 7942 consensus_queue.cc:1035] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:14.065068 8626 consensus_queue.cc:1035] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:14.089952 4300 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:14.097175 8594 tablet.cc:1722] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:14.097772 8594 tablet.cc:1620] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:14.101018 8676 tablet.cc:1722] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:14.102030 8676 tablet.cc:1620] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:14.104928 8674 tablet.cc:1722] T 6af430a9878a4a86b9057f94e3893510 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:14.105384 8674 tablet.cc:1620] T 6af430a9878a4a86b9057f94e3893510 P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:14.107498 4292 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=04480d680768439098c8518baa3d63ee] alter complete (version 2)
I20240502 21:27:14.189101 4291 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:14.197816 4422 tablet_service.cc:1467] Processing CreateTablet for tablet 81db5e77d00446cba3eb8345ee3df476 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=e1c7649d4d1248839d2699012aea3ee5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:14.197796 7193 tablet_service.cc:1467] Processing CreateTablet for tablet 81db5e77d00446cba3eb8345ee3df476 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=e1c7649d4d1248839d2699012aea3ee5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:14.198240 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81db5e77d00446cba3eb8345ee3df476. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:14.198266 7193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81db5e77d00446cba3eb8345ee3df476. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:14.199429 4555 tablet_service.cc:1467] Processing CreateTablet for tablet c0bc7002249e41079c92eb2364f870ab (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=e1c7649d4d1248839d2699012aea3ee5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:14.199908 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0bc7002249e41079c92eb2364f870ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:14.200039 7597 tablet_service.cc:1467] Processing CreateTablet for tablet c0bc7002249e41079c92eb2364f870ab (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=e1c7649d4d1248839d2699012aea3ee5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:14.200518 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0bc7002249e41079c92eb2364f870ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:14.203164 7195 tablet_service.cc:1467] Processing CreateTablet for tablet a507a60fb2094bd3b9dc24bec06db867 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=e1c7649d4d1248839d2699012aea3ee5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:14.203563 7195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a507a60fb2094bd3b9dc24bec06db867. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:14.204321 8612 tablet_bootstrap.cc:492] T 81db5e77d00446cba3eb8345ee3df476 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:14.205089 4818 tablet_service.cc:1467] Processing CreateTablet for tablet a507a60fb2094bd3b9dc24bec06db867 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=e1c7649d4d1248839d2699012aea3ee5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:14.205515 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a507a60fb2094bd3b9dc24bec06db867. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:14.206106 8612 tablet_bootstrap.cc:654] T 81db5e77d00446cba3eb8345ee3df476 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:14.207828 7463 tablet_service.cc:1467] Processing CreateTablet for tablet 81db5e77d00446cba3eb8345ee3df476 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=e1c7649d4d1248839d2699012aea3ee5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:14.207918 7194 tablet_service.cc:1467] Processing CreateTablet for tablet c0bc7002249e41079c92eb2364f870ab (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=e1c7649d4d1248839d2699012aea3ee5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:14.207947 7462 tablet_service.cc:1467] Processing CreateTablet for tablet a507a60fb2094bd3b9dc24bec06db867 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=e1c7649d4d1248839d2699012aea3ee5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:14.208343 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0bc7002249e41079c92eb2364f870ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:14.208861 7463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81db5e77d00446cba3eb8345ee3df476. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:14.211496 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a507a60fb2094bd3b9dc24bec06db867. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:14.218132 8612 tablet_bootstrap.cc:492] T 81db5e77d00446cba3eb8345ee3df476 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:14.218366 8612 ts_tablet_manager.cc:1367] T 81db5e77d00446cba3eb8345ee3df476 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20240502 21:27:14.219668 8612 raft_consensus.cc:348] T 81db5e77d00446cba3eb8345ee3df476 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:14.219950 8612 raft_consensus.cc:374] T 81db5e77d00446cba3eb8345ee3df476 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:14.221500 8612 raft_consensus.cc:729] T 81db5e77d00446cba3eb8345ee3df476 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:14.221909 8612 consensus_queue.cc:260] T 81db5e77d00446cba3eb8345ee3df476 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:14.223053 8612 ts_tablet_manager.cc:1398] T 81db5e77d00446cba3eb8345ee3df476 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:27:14.231843 8612 tablet_bootstrap.cc:492] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:14.232125 8683 tablet_bootstrap.cc:492] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:14.233536 8612 tablet_bootstrap.cc:654] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:14.235097 8683 tablet_bootstrap.cc:654] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:14.238847 8634 tablet_bootstrap.cc:492] T a507a60fb2094bd3b9dc24bec06db867 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:14.240592 8634 tablet_bootstrap.cc:654] T a507a60fb2094bd3b9dc24bec06db867 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:14.242808 4300 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:14.244921 8634 tablet_bootstrap.cc:492] T a507a60fb2094bd3b9dc24bec06db867 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:14.245172 8634 ts_tablet_manager.cc:1367] T a507a60fb2094bd3b9dc24bec06db867 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240502 21:27:14.246249 8634 raft_consensus.cc:348] T a507a60fb2094bd3b9dc24bec06db867 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:14.246675 8634 raft_consensus.cc:374] T a507a60fb2094bd3b9dc24bec06db867 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:14.246841 8634 raft_consensus.cc:729] T a507a60fb2094bd3b9dc24bec06db867 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:14.247279 8680 tablet_bootstrap.cc:492] T c0bc7002249e41079c92eb2364f870ab P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:14.247172 8634 consensus_queue.cc:260] T a507a60fb2094bd3b9dc24bec06db867 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:14.247859 8634 ts_tablet_manager.cc:1398] T a507a60fb2094bd3b9dc24bec06db867 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:14.248143 8634 tablet_bootstrap.cc:492] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:14.249980 8680 tablet_bootstrap.cc:654] T c0bc7002249e41079c92eb2364f870ab P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:14.250236 8634 tablet_bootstrap.cc:654] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:14.255450 8688 tablet_bootstrap.cc:492] T a507a60fb2094bd3b9dc24bec06db867 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:14.256114 8634 tablet_bootstrap.cc:492] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:14.257041 8634 ts_tablet_manager.cc:1367] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240502 21:27:14.257211 8686 tablet_bootstrap.cc:492] T c0bc7002249e41079c92eb2364f870ab P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:14.257892 8634 raft_consensus.cc:348] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:14.258116 8634 raft_consensus.cc:374] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:14.258204 8634 raft_consensus.cc:729] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:14.260035 8688 tablet_bootstrap.cc:654] T a507a60fb2094bd3b9dc24bec06db867 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:14.260308 8634 consensus_queue.cc:260] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:14.261111 8686 tablet_bootstrap.cc:654] T c0bc7002249e41079c92eb2364f870ab P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:14.261924 8634 ts_tablet_manager.cc:1398] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:27:14.262678 8594 tablet.cc:1722] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:14.268236 8688 tablet_bootstrap.cc:492] T a507a60fb2094bd3b9dc24bec06db867 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:14.268471 8688 ts_tablet_manager.cc:1367] T a507a60fb2094bd3b9dc24bec06db867 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240502 21:27:14.268927 8674 tablet.cc:1722] T 6af430a9878a4a86b9057f94e3893510 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:14.269415 8688 raft_consensus.cc:348] T a507a60fb2094bd3b9dc24bec06db867 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:14.269722 8688 raft_consensus.cc:374] T a507a60fb2094bd3b9dc24bec06db867 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:14.269876 8688 raft_consensus.cc:729] T a507a60fb2094bd3b9dc24bec06db867 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:14.270421 8676 tablet.cc:1722] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:14.270712 8688 consensus_queue.cc:260] T a507a60fb2094bd3b9dc24bec06db867 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:14.271356 8688 ts_tablet_manager.cc:1398] T a507a60fb2094bd3b9dc24bec06db867 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:14.271977 8680 tablet_bootstrap.cc:492] T c0bc7002249e41079c92eb2364f870ab P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:14.272131 8680 ts_tablet_manager.cc:1367] T c0bc7002249e41079c92eb2364f870ab P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20240502 21:27:14.272924 8680 raft_consensus.cc:348] T c0bc7002249e41079c92eb2364f870ab P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:14.273136 8680 raft_consensus.cc:374] T c0bc7002249e41079c92eb2364f870ab P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:14.273221 8680 raft_consensus.cc:729] T c0bc7002249e41079c92eb2364f870ab P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:14.273432 8680 consensus_queue.cc:260] T c0bc7002249e41079c92eb2364f870ab P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:14.273986 8680 ts_tablet_manager.cc:1398] T c0bc7002249e41079c92eb2364f870ab P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:14.277259 8612 tablet_bootstrap.cc:492] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:14.277413 8612 ts_tablet_manager.cc:1367] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.046s user 0.006s sys 0.004s
I20240502 21:27:14.278183 8612 raft_consensus.cc:348] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:14.278918 8612 raft_consensus.cc:374] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:14.279088 8612 raft_consensus.cc:729] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:14.279373 8612 consensus_queue.cc:260] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:14.280133 8612 ts_tablet_manager.cc:1398] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:14.280483 8612 tablet_bootstrap.cc:492] T c0bc7002249e41079c92eb2364f870ab P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:14.282599 8612 tablet_bootstrap.cc:654] T c0bc7002249e41079c92eb2364f870ab P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:14.290685 8686 tablet_bootstrap.cc:492] T c0bc7002249e41079c92eb2364f870ab P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:14.290841 8612 tablet_bootstrap.cc:492] T c0bc7002249e41079c92eb2364f870ab P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:14.290957 8686 ts_tablet_manager.cc:1367] T c0bc7002249e41079c92eb2364f870ab P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.006s
I20240502 21:27:14.291080 8612 ts_tablet_manager.cc:1367] T c0bc7002249e41079c92eb2364f870ab P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.001s
I20240502 21:27:14.291867 8612 raft_consensus.cc:348] T c0bc7002249e41079c92eb2364f870ab P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:14.292033 8683 tablet_bootstrap.cc:492] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:14.292084 8612 raft_consensus.cc:374] T c0bc7002249e41079c92eb2364f870ab P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:14.292148 8612 raft_consensus.cc:729] T c0bc7002249e41079c92eb2364f870ab P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:14.292189 8683 ts_tablet_manager.cc:1367] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.060s user 0.006s sys 0.003s
I20240502 21:27:14.292443 8612 consensus_queue.cc:260] T c0bc7002249e41079c92eb2364f870ab P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:14.292984 8612 ts_tablet_manager.cc:1398] T c0bc7002249e41079c92eb2364f870ab P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:14.292953 8683 raft_consensus.cc:348] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:14.293150 8683 raft_consensus.cc:374] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:14.293236 8683 raft_consensus.cc:729] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:14.293478 8683 consensus_queue.cc:260] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:14.293979 8683 ts_tablet_manager.cc:1398] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:14.294271 4298 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=04480d680768439098c8518baa3d63ee] alter complete (version 3)
I20240502 21:27:14.295354 8679 raft_consensus.cc:482] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:14.295498 8679 raft_consensus.cc:504] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:14.296406 8679 leader_election.cc:288] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:14.296823 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a507a60fb2094bd3b9dc24bec06db867" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:27:14.296940 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a507a60fb2094bd3b9dc24bec06db867" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:14.297075 4838 raft_consensus.cc:2454] T a507a60fb2094bd3b9dc24bec06db867 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:14.297173 7484 raft_consensus.cc:2454] T a507a60fb2094bd3b9dc24bec06db867 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:14.297520 7148 leader_election.cc:302] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:14.297983 8679 raft_consensus.cc:2781] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:14.298106 8679 raft_consensus.cc:482] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:14.298223 8679 raft_consensus.cc:3037] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:14.300603 8679 raft_consensus.cc:504] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:14.301134 8679 leader_election.cc:288] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:14.301642 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a507a60fb2094bd3b9dc24bec06db867" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:27:14.301818 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a507a60fb2094bd3b9dc24bec06db867" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:14.301908 4838 raft_consensus.cc:3037] T a507a60fb2094bd3b9dc24bec06db867 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:14.302024 7484 raft_consensus.cc:3037] T a507a60fb2094bd3b9dc24bec06db867 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:14.304450 7484 raft_consensus.cc:2454] T a507a60fb2094bd3b9dc24bec06db867 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:14.307265 4838 raft_consensus.cc:2454] T a507a60fb2094bd3b9dc24bec06db867 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:14.307801 7148 leader_election.cc:302] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:14.308185 8679 raft_consensus.cc:2781] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:14.308445 8679 raft_consensus.cc:686] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:27:14.308919 8679 consensus_queue.cc:237] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:14.311470 8686 raft_consensus.cc:348] T c0bc7002249e41079c92eb2364f870ab P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:14.312507 8686 raft_consensus.cc:374] T c0bc7002249e41079c92eb2364f870ab P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:14.312618 8686 raft_consensus.cc:729] T c0bc7002249e41079c92eb2364f870ab P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:14.312968 8686 consensus_queue.cc:260] T c0bc7002249e41079c92eb2364f870ab P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:14.315033 8686 ts_tablet_manager.cc:1398] T c0bc7002249e41079c92eb2364f870ab P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240502 21:27:14.326658 4291 catalog_manager.cc:5484] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69). New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:14.406098 4291 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:14.406463 4291 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:14.410183 4291 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 6af430a9878a4a86b9057f94e3893510
I20240502 21:27:14.411084 7194 tablet_service.cc:1514] Processing DeleteTablet for tablet 6af430a9878a4a86b9057f94e3893510 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:14 UTC) from {username='slave'} at 127.0.0.1:44652
I20240502 21:27:14.411300 7328 tablet_service.cc:1514] Processing DeleteTablet for tablet 6af430a9878a4a86b9057f94e3893510 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:14 UTC) from {username='slave'} at 127.0.0.1:47078
I20240502 21:27:14.411913 8696 tablet_replica.cc:330] T 6af430a9878a4a86b9057f94e3893510 P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:27:14.411993 7463 tablet_service.cc:1514] Processing DeleteTablet for tablet 6af430a9878a4a86b9057f94e3893510 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:14 UTC) from {username='slave'} at 127.0.0.1:37182
I20240502 21:27:14.412153 8696 raft_consensus.cc:2229] T 6af430a9878a4a86b9057f94e3893510 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:14.412391 8696 raft_consensus.cc:2258] T 6af430a9878a4a86b9057f94e3893510 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:14.411787 8695 tablet_replica.cc:330] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:27:14.412709 8697 tablet_replica.cc:330] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:27:14.412766 8695 raft_consensus.cc:2229] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:14.412925 8697 raft_consensus.cc:2229] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:14.413254 8695 raft_consensus.cc:2258] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:14.413251 8697 raft_consensus.cc:2258] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:14.413427 8696 ts_tablet_manager.cc:1875] T 6af430a9878a4a86b9057f94e3893510 P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:14.414325 8695 ts_tablet_manager.cc:1875] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:14.414438 8697 ts_tablet_manager.cc:1875] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:14.418634 8696 ts_tablet_manager.cc:1888] T 6af430a9878a4a86b9057f94e3893510 P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:14.418814 8696 log.cc:1196] T 6af430a9878a4a86b9057f94e3893510 P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/6af430a9878a4a86b9057f94e3893510
I20240502 21:27:14.419309 8696 ts_tablet_manager.cc:1909] T 6af430a9878a4a86b9057f94e3893510 P d0456c5e55a542128688904e003d403c: Deleting consensus metadata
I20240502 21:27:14.423112 8695 ts_tablet_manager.cc:1888] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:14.423308 8695 log.cc:1196] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/6af430a9878a4a86b9057f94e3893510
I20240502 21:27:14.423931 8695 ts_tablet_manager.cc:1909] T 6af430a9878a4a86b9057f94e3893510 P 39981a8dfe94462b9a73ba489af4c714: Deleting consensus metadata
I20240502 21:27:14.425249 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet 6af430a9878a4a86b9057f94e3893510 (table rebalancer_dynamic_table_new_name [id=04480d680768439098c8518baa3d63ee]) successfully deleted
I20240502 21:27:14.425696 8697 ts_tablet_manager.cc:1888] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:14.425817 8697 log.cc:1196] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/6af430a9878a4a86b9057f94e3893510
I20240502 21:27:14.426187 8697 ts_tablet_manager.cc:1909] T 6af430a9878a4a86b9057f94e3893510 P 31a72286397c4edc9a63eca3412be168: Deleting consensus metadata
I20240502 21:27:14.427287 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet 6af430a9878a4a86b9057f94e3893510 (table rebalancer_dynamic_table_new_name [id=04480d680768439098c8518baa3d63ee]) successfully deleted
I20240502 21:27:14.427366 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet 6af430a9878a4a86b9057f94e3893510 (table rebalancer_dynamic_table_new_name [id=04480d680768439098c8518baa3d63ee]) successfully deleted
I20240502 21:27:14.485525 8654 raft_consensus.cc:482] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:14.485739 8654 raft_consensus.cc:504] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:14.486593 8654 leader_election.cc:288] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:14.487164 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81db5e77d00446cba3eb8345ee3df476" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:14.487407 7214 raft_consensus.cc:2454] T 81db5e77d00446cba3eb8345ee3df476 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:14.487505 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81db5e77d00446cba3eb8345ee3df476" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:14.487831 7484 raft_consensus.cc:2454] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:14.488312 4377 leader_election.cc:302] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:14.488618 8654 raft_consensus.cc:2781] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:14.488756 8654 raft_consensus.cc:482] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:14.488858 8654 raft_consensus.cc:3037] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:14.491797 8654 raft_consensus.cc:504] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:14.492439 8654 leader_election.cc:288] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:14.492835 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81db5e77d00446cba3eb8345ee3df476" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:14.493026 7214 raft_consensus.cc:3037] T 81db5e77d00446cba3eb8345ee3df476 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:14.492998 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81db5e77d00446cba3eb8345ee3df476" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:14.493286 7484 raft_consensus.cc:3037] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:14.496021 7214 raft_consensus.cc:2454] T 81db5e77d00446cba3eb8345ee3df476 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:14.496771 4377 leader_election.cc:302] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:14.497138 8654 raft_consensus.cc:2781] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:14.497308 8654 raft_consensus.cc:686] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:27:14.497642 8654 consensus_queue.cc:237] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:14.499356 7484 raft_consensus.cc:2454] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:14.502175 4291 catalog_manager.cc:5484] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:27:14.512784 4291 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:14.519886 8661 raft_consensus.cc:482] T c0bc7002249e41079c92eb2364f870ab P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:14.520057 8661 raft_consensus.cc:504] T c0bc7002249e41079c92eb2364f870ab P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:14.520865 8661 leader_election.cc:288] T c0bc7002249e41079c92eb2364f870ab P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:14.521701 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 19c14a92c89e433a83990d05d7c79409 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cb9d694cb18d4ba6960c1bf712c42e40]), partition=
I20240502 21:27:14.522176 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19c14a92c89e433a83990d05d7c79409. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:14.522256 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0bc7002249e41079c92eb2364f870ab" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:14.522560 7618 raft_consensus.cc:2454] T c0bc7002249e41079c92eb2364f870ab P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:27:14.523181 4509 leader_election.cc:302] T c0bc7002249e41079c92eb2364f870ab P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:27:14.523532 8661 raft_consensus.cc:2781] T c0bc7002249e41079c92eb2364f870ab P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:14.523698 8661 raft_consensus.cc:482] T c0bc7002249e41079c92eb2364f870ab P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:14.523820 8661 raft_consensus.cc:3037] T c0bc7002249e41079c92eb2364f870ab P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:14.527439 8661 raft_consensus.cc:504] T c0bc7002249e41079c92eb2364f870ab P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:14.528558 8661 leader_election.cc:288] T c0bc7002249e41079c92eb2364f870ab P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Requested vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:14.528777 4818 tablet_service.cc:1467] Processing CreateTablet for tablet 19c14a92c89e433a83990d05d7c79409 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cb9d694cb18d4ba6960c1bf712c42e40]), partition=
I20240502 21:27:14.528796 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0bc7002249e41079c92eb2364f870ab" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:14.529028 7618 raft_consensus.cc:3037] T c0bc7002249e41079c92eb2364f870ab P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:14.529249 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19c14a92c89e433a83990d05d7c79409. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:14.531026 7618 raft_consensus.cc:2454] T c0bc7002249e41079c92eb2364f870ab P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:27:14.531452 4509 leader_election.cc:302] T c0bc7002249e41079c92eb2364f870ab P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:27:14.531873 8661 raft_consensus.cc:2781] T c0bc7002249e41079c92eb2364f870ab P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:14.532653 8661 raft_consensus.cc:686] T c0bc7002249e41079c92eb2364f870ab P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:27:14.533033 8661 consensus_queue.cc:237] T c0bc7002249e41079c92eb2364f870ab P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:14.539229 7194 tablet_service.cc:1467] Processing CreateTablet for tablet 19c14a92c89e433a83990d05d7c79409 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cb9d694cb18d4ba6960c1bf712c42e40]), partition=
I20240502 21:27:14.539659 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19c14a92c89e433a83990d05d7c79409. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:14.546066 8612 tablet_bootstrap.cc:492] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:14.548283 8612 tablet_bootstrap.cc:654] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:14.550734 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0bc7002249e41079c92eb2364f870ab" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:14.550849 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0bc7002249e41079c92eb2364f870ab" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:14.551013 7214 raft_consensus.cc:2454] T c0bc7002249e41079c92eb2364f870ab P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:27:14.551255 4300 catalog_manager.cc:5484] T c0bc7002249e41079c92eb2364f870ab P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:14.555239 8612 tablet_bootstrap.cc:492] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:14.555394 8612 ts_tablet_manager.cc:1367] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240502 21:27:14.556244 8612 raft_consensus.cc:348] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:14.556419 8612 raft_consensus.cc:374] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:14.556515 8612 raft_consensus.cc:729] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:14.556743 8612 consensus_queue.cc:260] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:14.557443 8612 ts_tablet_manager.cc:1398] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:14.566071 8688 tablet_bootstrap.cc:492] T 19c14a92c89e433a83990d05d7c79409 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:14.567389 8688 tablet_bootstrap.cc:654] T 19c14a92c89e433a83990d05d7c79409 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:14.570981 8688 tablet_bootstrap.cc:492] T 19c14a92c89e433a83990d05d7c79409 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:14.571130 8688 ts_tablet_manager.cc:1367] T 19c14a92c89e433a83990d05d7c79409 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:27:14.571777 8688 raft_consensus.cc:348] T 19c14a92c89e433a83990d05d7c79409 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:14.571952 8688 raft_consensus.cc:374] T 19c14a92c89e433a83990d05d7c79409 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:14.572031 8688 raft_consensus.cc:729] T 19c14a92c89e433a83990d05d7c79409 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:14.572304 8688 consensus_queue.cc:260] T 19c14a92c89e433a83990d05d7c79409 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:14.572877 8688 ts_tablet_manager.cc:1398] T 19c14a92c89e433a83990d05d7c79409 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:14.575935 8704 tablet_bootstrap.cc:492] T 19c14a92c89e433a83990d05d7c79409 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:14.578078 8704 tablet_bootstrap.cc:654] T 19c14a92c89e433a83990d05d7c79409 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:14.580639 8704 tablet_bootstrap.cc:492] T 19c14a92c89e433a83990d05d7c79409 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:27:14.580816 8704 ts_tablet_manager.cc:1367] T 19c14a92c89e433a83990d05d7c79409 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 21:27:14.581810 8704 raft_consensus.cc:348] T 19c14a92c89e433a83990d05d7c79409 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:14.582099 8704 raft_consensus.cc:374] T 19c14a92c89e433a83990d05d7c79409 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:14.582217 8704 raft_consensus.cc:729] T 19c14a92c89e433a83990d05d7c79409 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:14.582515 8704 consensus_queue.cc:260] T 19c14a92c89e433a83990d05d7c79409 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:14.583423 8704 ts_tablet_manager.cc:1398] T 19c14a92c89e433a83990d05d7c79409 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:14.677064 8649 raft_consensus.cc:482] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:14.677333 8649 raft_consensus.cc:504] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:14.678380 8649 leader_election.cc:288] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:14.678897 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c14a92c89e433a83990d05d7c79409" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:27:14.679150 4838 raft_consensus.cc:2454] T 19c14a92c89e433a83990d05d7c79409 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:14.679719 7148 leader_election.cc:302] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:14.681921 8649 raft_consensus.cc:2781] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:14.682098 8649 raft_consensus.cc:482] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:14.682296 8649 raft_consensus.cc:3037] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:14.686574 8649 raft_consensus.cc:504] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:14.688086 8649 leader_election.cc:288] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:14.688194 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c14a92c89e433a83990d05d7c79409" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:27:14.688442 4838 raft_consensus.cc:3037] T 19c14a92c89e433a83990d05d7c79409 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:14.688915 4704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c14a92c89e433a83990d05d7c79409" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:27:14.689167 4704 raft_consensus.cc:2454] T 19c14a92c89e433a83990d05d7c79409 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:14.689105 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c14a92c89e433a83990d05d7c79409" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:27:14.689347 4706 raft_consensus.cc:3037] T 19c14a92c89e433a83990d05d7c79409 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:14.690757 4838 raft_consensus.cc:2454] T 19c14a92c89e433a83990d05d7c79409 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:14.691841 7148 leader_election.cc:302] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:14.692132 8679 raft_consensus.cc:2781] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:14.692291 8679 raft_consensus.cc:686] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:27:14.692545 8679 consensus_queue.cc:237] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:14.692965 4706 raft_consensus.cc:2454] T 19c14a92c89e433a83990d05d7c79409 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:14.697527 4291 catalog_manager.cc:5484] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69). New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:14.713732 8679 consensus_queue.cc:1035] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:14.718243 8679 consensus_queue.cc:1035] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:14.813005 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:14.819483 4838 raft_consensus.cc:1261] T 19c14a92c89e433a83990d05d7c79409 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:14.820497 8679 consensus_queue.cc:1035] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:14.821453 4706 raft_consensus.cc:1261] T 19c14a92c89e433a83990d05d7c79409 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:14.822479 8651 consensus_queue.cc:1035] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:14.829635 4294 catalog_manager.cc:5576] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537) does not have the latest schema for tablet 19c14a92c89e433a83990d05d7c79409 (table rebalancer_dynamic_table [id=cb9d694cb18d4ba6960c1bf712c42e40]). Expected version 1 got 0
I20240502 21:27:14.840452 8715 tablet.cc:1722] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:14.841030 8715 tablet.cc:1620] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:14.851009 4291 catalog_manager.cc:5576] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537) does not have the latest schema for tablet 19c14a92c89e433a83990d05d7c79409 (table rebalancer_dynamic_table [id=cb9d694cb18d4ba6960c1bf712c42e40]). Expected version 1 got 0
I20240502 21:27:14.852401 4294 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=cb9d694cb18d4ba6960c1bf712c42e40] alter complete (version 1)
I20240502 21:27:14.847997 8716 tablet.cc:1722] T 19c14a92c89e433a83990d05d7c79409 P 398c3c8f3195492c900439ad5e1931bb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:14.853703 8716 tablet.cc:1620] T 19c14a92c89e433a83990d05d7c79409 P 398c3c8f3195492c900439ad5e1931bb: MemRowSet was empty: no flush needed.
I20240502 21:27:14.857062 4291 catalog_manager.cc:5576] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449) does not have the latest schema for tablet 19c14a92c89e433a83990d05d7c79409 (table rebalancer_dynamic_table [id=cb9d694cb18d4ba6960c1bf712c42e40]). Expected version 1 got 0
I20240502 21:27:14.868592 8717 tablet.cc:1722] T 19c14a92c89e433a83990d05d7c79409 P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:14.869372 8717 tablet.cc:1620] T 19c14a92c89e433a83990d05d7c79409 P 73a1f2ca1f544de9b0e3b37015f85328: MemRowSet was empty: no flush needed.
I20240502 21:27:14.870189 4291 catalog_manager.cc:5576] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449) does not have the latest schema for tablet 19c14a92c89e433a83990d05d7c79409 (table rebalancer_dynamic_table [id=cb9d694cb18d4ba6960c1bf712c42e40]). Expected version 1 got 0
I20240502 21:27:14.886018 8582 consensus_queue.cc:1035] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:14.892025 8654 consensus_queue.cc:1035] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:14.928872 7215 raft_consensus.cc:3037] T c0bc7002249e41079c92eb2364f870ab P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:14.931912 8661 consensus_queue.cc:1035] T c0bc7002249e41079c92eb2364f870ab P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:14.938097 8661 consensus_queue.cc:1035] T c0bc7002249e41079c92eb2364f870ab P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:14.959079 4291 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:14.967658 8715 tablet.cc:1722] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:14.968187 8715 tablet.cc:1620] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:14.968731 8716 tablet.cc:1722] T 19c14a92c89e433a83990d05d7c79409 P 398c3c8f3195492c900439ad5e1931bb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:14.969225 8716 tablet.cc:1620] T 19c14a92c89e433a83990d05d7c79409 P 398c3c8f3195492c900439ad5e1931bb: MemRowSet was empty: no flush needed.
I20240502 21:27:14.969457 8717 tablet.cc:1722] T 19c14a92c89e433a83990d05d7c79409 P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:14.969962 8717 tablet.cc:1620] T 19c14a92c89e433a83990d05d7c79409 P 73a1f2ca1f544de9b0e3b37015f85328: MemRowSet was empty: no flush needed.
I20240502 21:27:14.979545 4291 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=cb9d694cb18d4ba6960c1bf712c42e40] alter complete (version 2)
I20240502 21:27:15.059037 4294 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:15.069881 4555 tablet_service.cc:1467] Processing CreateTablet for tablet 8ffa252784984df1bbb0ce35a26d4865 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=57ddb6dad3c444239d35a1ed0119bba9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:15.070416 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ffa252784984df1bbb0ce35a26d4865. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:15.070425 7597 tablet_service.cc:1467] Processing CreateTablet for tablet 8ffa252784984df1bbb0ce35a26d4865 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=57ddb6dad3c444239d35a1ed0119bba9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:15.070856 7596 tablet_service.cc:1467] Processing CreateTablet for tablet a8a3d57ff2184ab3bc09140afbffe0b8 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=57ddb6dad3c444239d35a1ed0119bba9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:15.071389 7328 tablet_service.cc:1467] Processing CreateTablet for tablet 8ffa252784984df1bbb0ce35a26d4865 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=57ddb6dad3c444239d35a1ed0119bba9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:15.071919 7595 tablet_service.cc:1467] Processing CreateTablet for tablet 37d864bed6ef46e4a3f57de0373bbe57 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=57ddb6dad3c444239d35a1ed0119bba9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:15.073542 4422 tablet_service.cc:1467] Processing CreateTablet for tablet 37d864bed6ef46e4a3f57de0373bbe57 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=57ddb6dad3c444239d35a1ed0119bba9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:15.074019 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37d864bed6ef46e4a3f57de0373bbe57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:15.074052 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ffa252784984df1bbb0ce35a26d4865. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:15.078027 8726 tablet_bootstrap.cc:492] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:15.078066 4686 tablet_service.cc:1467] Processing CreateTablet for tablet a8a3d57ff2184ab3bc09140afbffe0b8 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=57ddb6dad3c444239d35a1ed0119bba9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:15.078510 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a3d57ff2184ab3bc09140afbffe0b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:15.070955 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ffa252784984df1bbb0ce35a26d4865. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:15.079491 7596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a3d57ff2184ab3bc09140afbffe0b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:15.080014 7595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37d864bed6ef46e4a3f57de0373bbe57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:15.080574 7194 tablet_service.cc:1467] Processing CreateTablet for tablet a8a3d57ff2184ab3bc09140afbffe0b8 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=57ddb6dad3c444239d35a1ed0119bba9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:15.080962 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a3d57ff2184ab3bc09140afbffe0b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:15.080909 7329 tablet_service.cc:1467] Processing CreateTablet for tablet 37d864bed6ef46e4a3f57de0373bbe57 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=57ddb6dad3c444239d35a1ed0119bba9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:15.081318 7329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37d864bed6ef46e4a3f57de0373bbe57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:15.081930 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:15.091790 8715 tablet.cc:1722] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:15.101007 8726 tablet_bootstrap.cc:654] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:15.101138 8716 tablet.cc:1722] T 19c14a92c89e433a83990d05d7c79409 P 398c3c8f3195492c900439ad5e1931bb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:15.111532 8717 tablet.cc:1722] T 19c14a92c89e433a83990d05d7c79409 P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:15.111912 8731 tablet_bootstrap.cc:492] T 37d864bed6ef46e4a3f57de0373bbe57 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:15.113425 4294 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=cb9d694cb18d4ba6960c1bf712c42e40] alter complete (version 3)
I20240502 21:27:15.115455 8731 tablet_bootstrap.cc:654] T 37d864bed6ef46e4a3f57de0373bbe57 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:15.116605 8728 tablet_bootstrap.cc:492] T a8a3d57ff2184ab3bc09140afbffe0b8 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:15.119038 8728 tablet_bootstrap.cc:654] T a8a3d57ff2184ab3bc09140afbffe0b8 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:15.120044 8729 tablet_bootstrap.cc:492] T 8ffa252784984df1bbb0ce35a26d4865 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:15.122267 8729 tablet_bootstrap.cc:654] T 8ffa252784984df1bbb0ce35a26d4865 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:15.129601 8732 tablet_bootstrap.cc:492] T 37d864bed6ef46e4a3f57de0373bbe57 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:15.130502 8729 tablet_bootstrap.cc:492] T 8ffa252784984df1bbb0ce35a26d4865 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:15.131112 8728 tablet_bootstrap.cc:492] T a8a3d57ff2184ab3bc09140afbffe0b8 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:27:15.131338 8728 ts_tablet_manager.cc:1367] T a8a3d57ff2184ab3bc09140afbffe0b8 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240502 21:27:15.131723 8726 tablet_bootstrap.cc:492] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:15.131891 8726 ts_tablet_manager.cc:1367] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.054s user 0.011s sys 0.000s
I20240502 21:27:15.131944 8729 ts_tablet_manager.cc:1367] T 8ffa252784984df1bbb0ce35a26d4865 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.004s
I20240502 21:27:15.132647 8732 tablet_bootstrap.cc:654] T 37d864bed6ef46e4a3f57de0373bbe57 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:15.132783 8726 raft_consensus.cc:348] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:15.133347 8728 raft_consensus.cc:348] T a8a3d57ff2184ab3bc09140afbffe0b8 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:15.134254 8728 raft_consensus.cc:374] T a8a3d57ff2184ab3bc09140afbffe0b8 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:15.133651 8729 raft_consensus.cc:348] T 8ffa252784984df1bbb0ce35a26d4865 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:15.134593 8728 raft_consensus.cc:729] T a8a3d57ff2184ab3bc09140afbffe0b8 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:15.134716 8727 tablet_bootstrap.cc:492] T a8a3d57ff2184ab3bc09140afbffe0b8 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:15.134761 8729 raft_consensus.cc:374] T 8ffa252784984df1bbb0ce35a26d4865 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:15.134981 8729 raft_consensus.cc:729] T 8ffa252784984df1bbb0ce35a26d4865 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:15.135345 8728 consensus_queue.cc:260] T a8a3d57ff2184ab3bc09140afbffe0b8 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:15.136281 8729 consensus_queue.cc:260] T 8ffa252784984df1bbb0ce35a26d4865 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:15.137012 8728 ts_tablet_manager.cc:1398] T a8a3d57ff2184ab3bc09140afbffe0b8 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 21:27:15.137452 8726 raft_consensus.cc:374] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:15.137584 8726 raft_consensus.cc:729] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:15.138388 8727 tablet_bootstrap.cc:654] T a8a3d57ff2184ab3bc09140afbffe0b8 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:15.138767 8729 ts_tablet_manager.cc:1398] T 8ffa252784984df1bbb0ce35a26d4865 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 21:27:15.139257 8729 tablet_bootstrap.cc:492] T a8a3d57ff2184ab3bc09140afbffe0b8 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:15.140661 8726 consensus_queue.cc:260] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:15.141455 8726 ts_tablet_manager.cc:1398] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 21:27:15.142350 8729 tablet_bootstrap.cc:654] T a8a3d57ff2184ab3bc09140afbffe0b8 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:15.142904 8732 tablet_bootstrap.cc:492] T 37d864bed6ef46e4a3f57de0373bbe57 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:15.143097 8732 ts_tablet_manager.cc:1367] T 37d864bed6ef46e4a3f57de0373bbe57 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.002s
I20240502 21:27:15.144863 8732 raft_consensus.cc:348] T 37d864bed6ef46e4a3f57de0373bbe57 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:15.145169 8732 raft_consensus.cc:374] T 37d864bed6ef46e4a3f57de0373bbe57 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:15.145339 8732 raft_consensus.cc:729] T 37d864bed6ef46e4a3f57de0373bbe57 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:15.145604 8732 consensus_queue.cc:260] T 37d864bed6ef46e4a3f57de0373bbe57 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:15.146134 8732 ts_tablet_manager.cc:1398] T 37d864bed6ef46e4a3f57de0373bbe57 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240502 21:27:15.153146 8729 tablet_bootstrap.cc:492] T a8a3d57ff2184ab3bc09140afbffe0b8 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:15.153326 8729 ts_tablet_manager.cc:1367] T a8a3d57ff2184ab3bc09140afbffe0b8 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.005s
I20240502 21:27:15.154139 8729 raft_consensus.cc:348] T a8a3d57ff2184ab3bc09140afbffe0b8 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:15.154387 8729 raft_consensus.cc:374] T a8a3d57ff2184ab3bc09140afbffe0b8 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:15.154464 8729 raft_consensus.cc:729] T a8a3d57ff2184ab3bc09140afbffe0b8 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:15.154781 8729 consensus_queue.cc:260] T a8a3d57ff2184ab3bc09140afbffe0b8 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:15.156486 8729 ts_tablet_manager.cc:1398] T a8a3d57ff2184ab3bc09140afbffe0b8 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240502 21:27:15.156843 8729 tablet_bootstrap.cc:492] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:15.159289 8729 tablet_bootstrap.cc:654] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:15.167974 8731 tablet_bootstrap.cc:492] T 37d864bed6ef46e4a3f57de0373bbe57 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:15.168313 8731 ts_tablet_manager.cc:1367] T 37d864bed6ef46e4a3f57de0373bbe57 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.057s user 0.010s sys 0.000s
I20240502 21:27:15.169675 8731 raft_consensus.cc:348] T 37d864bed6ef46e4a3f57de0373bbe57 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:15.170053 8731 raft_consensus.cc:374] T 37d864bed6ef46e4a3f57de0373bbe57 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:15.170255 8731 raft_consensus.cc:729] T 37d864bed6ef46e4a3f57de0373bbe57 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:15.170514 8731 consensus_queue.cc:260] T 37d864bed6ef46e4a3f57de0373bbe57 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:15.172101 8727 tablet_bootstrap.cc:492] T a8a3d57ff2184ab3bc09140afbffe0b8 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:15.172313 8727 ts_tablet_manager.cc:1367] T a8a3d57ff2184ab3bc09140afbffe0b8 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.038s user 0.026s sys 0.004s
I20240502 21:27:15.173266 8731 ts_tablet_manager.cc:1398] T 37d864bed6ef46e4a3f57de0373bbe57 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:27:15.173630 8731 tablet_bootstrap.cc:492] T 8ffa252784984df1bbb0ce35a26d4865 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:15.173498 8727 raft_consensus.cc:348] T a8a3d57ff2184ab3bc09140afbffe0b8 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:15.173723 8727 raft_consensus.cc:374] T a8a3d57ff2184ab3bc09140afbffe0b8 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:15.173833 8727 raft_consensus.cc:729] T a8a3d57ff2184ab3bc09140afbffe0b8 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:15.174095 8727 consensus_queue.cc:260] T a8a3d57ff2184ab3bc09140afbffe0b8 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:15.174674 8729 tablet_bootstrap.cc:492] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:15.174814 8727 ts_tablet_manager.cc:1398] T a8a3d57ff2184ab3bc09140afbffe0b8 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:15.174921 8729 ts_tablet_manager.cc:1367] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.008s
I20240502 21:27:15.175669 8729 raft_consensus.cc:348] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:15.175940 8729 raft_consensus.cc:374] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:15.176106 8729 raft_consensus.cc:729] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:15.176430 8729 consensus_queue.cc:260] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:15.178094 8731 tablet_bootstrap.cc:654] T 8ffa252784984df1bbb0ce35a26d4865 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:15.178192 8729 ts_tablet_manager.cc:1398] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:15.180123 8731 tablet_bootstrap.cc:492] T 8ffa252784984df1bbb0ce35a26d4865 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:15.180193 7942 raft_consensus.cc:482] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:15.180399 8731 ts_tablet_manager.cc:1367] T 8ffa252784984df1bbb0ce35a26d4865 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 21:27:15.180342 7942 raft_consensus.cc:504] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:15.181042 7942 leader_election.cc:288] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:15.181475 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d864bed6ef46e4a3f57de0373bbe57" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:15.181546 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d864bed6ef46e4a3f57de0373bbe57" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:27:15.181676 7350 raft_consensus.cc:2454] T 37d864bed6ef46e4a3f57de0373bbe57 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:15.181797 4443 raft_consensus.cc:2454] T 37d864bed6ef46e4a3f57de0373bbe57 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:15.182060 7551 leader_election.cc:302] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:15.182293 7942 raft_consensus.cc:2781] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:15.182390 7942 raft_consensus.cc:482] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:15.182463 7942 raft_consensus.cc:3037] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:15.184230 8731 raft_consensus.cc:348] T 8ffa252784984df1bbb0ce35a26d4865 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:15.184523 8731 raft_consensus.cc:374] T 8ffa252784984df1bbb0ce35a26d4865 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:15.184616 8731 raft_consensus.cc:729] T 8ffa252784984df1bbb0ce35a26d4865 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:15.184852 8731 consensus_queue.cc:260] T 8ffa252784984df1bbb0ce35a26d4865 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:15.184954 7942 raft_consensus.cc:504] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:15.185369 8731 ts_tablet_manager.cc:1398] T 8ffa252784984df1bbb0ce35a26d4865 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 21:27:15.185597 7942 leader_election.cc:288] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:15.186012 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d864bed6ef46e4a3f57de0373bbe57" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:27:15.186231 4443 raft_consensus.cc:3037] T 37d864bed6ef46e4a3f57de0373bbe57 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:15.186254 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d864bed6ef46e4a3f57de0373bbe57" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:15.186455 7350 raft_consensus.cc:3037] T 37d864bed6ef46e4a3f57de0373bbe57 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:15.188858 4443 raft_consensus.cc:2454] T 37d864bed6ef46e4a3f57de0373bbe57 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:15.189255 7350 raft_consensus.cc:2454] T 37d864bed6ef46e4a3f57de0373bbe57 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:15.189422 7549 leader_election.cc:302] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:15.189743 7942 raft_consensus.cc:2781] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:15.190160 7942 raft_consensus.cc:686] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Becoming Leader. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Running, Role: LEADER
I20240502 21:27:15.190718 7942 consensus_queue.cc:237] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:15.194104 4298 catalog_manager.cc:5484] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ddd486109f5434cace13d60d633ac04 (127.29.82.72). New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:15.224109 4298 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:15.224331 4298 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:15.227643 4298 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 19c14a92c89e433a83990d05d7c79409
I20240502 21:27:15.228576 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet 19c14a92c89e433a83990d05d7c79409 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:15 UTC) from {username='slave'} at 127.0.0.1:44764
I20240502 21:27:15.228770 7194 tablet_service.cc:1514] Processing DeleteTablet for tablet 19c14a92c89e433a83990d05d7c79409 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:15 UTC) from {username='slave'} at 127.0.0.1:44652
I20240502 21:27:15.229879 4818 tablet_service.cc:1514] Processing DeleteTablet for tablet 19c14a92c89e433a83990d05d7c79409 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:15 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:27:15.238546 8745 tablet_replica.cc:330] T 19c14a92c89e433a83990d05d7c79409 P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:27:15.239205 8743 tablet_replica.cc:330] T 19c14a92c89e433a83990d05d7c79409 P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:27:15.239185 8745 raft_consensus.cc:2229] T 19c14a92c89e433a83990d05d7c79409 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:15.239467 8745 raft_consensus.cc:2258] T 19c14a92c89e433a83990d05d7c79409 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:15.239537 8743 raft_consensus.cc:2229] T 19c14a92c89e433a83990d05d7c79409 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:15.239838 8743 raft_consensus.cc:2258] T 19c14a92c89e433a83990d05d7c79409 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:15.240440 8745 ts_tablet_manager.cc:1875] T 19c14a92c89e433a83990d05d7c79409 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:15.240725 8743 ts_tablet_manager.cc:1875] T 19c14a92c89e433a83990d05d7c79409 P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:15.240793 8744 tablet_replica.cc:330] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:27:15.241044 8744 raft_consensus.cc:2229] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:15.241451 8744 raft_consensus.cc:2258] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:15.242357 8744 ts_tablet_manager.cc:1875] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:15.250263 8745 ts_tablet_manager.cc:1888] T 19c14a92c89e433a83990d05d7c79409 P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:15.250464 8745 log.cc:1196] T 19c14a92c89e433a83990d05d7c79409 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/19c14a92c89e433a83990d05d7c79409
I20240502 21:27:15.250887 8743 ts_tablet_manager.cc:1888] T 19c14a92c89e433a83990d05d7c79409 P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:15.251003 8745 ts_tablet_manager.cc:1909] T 19c14a92c89e433a83990d05d7c79409 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:27:15.251025 8743 log.cc:1196] T 19c14a92c89e433a83990d05d7c79409 P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/19c14a92c89e433a83990d05d7c79409
I20240502 21:27:15.251502 8743 ts_tablet_manager.cc:1909] T 19c14a92c89e433a83990d05d7c79409 P 398c3c8f3195492c900439ad5e1931bb: Deleting consensus metadata
I20240502 21:27:15.251724 8744 ts_tablet_manager.cc:1888] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:15.251852 8744 log.cc:1196] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/19c14a92c89e433a83990d05d7c79409
I20240502 21:27:15.252185 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet 19c14a92c89e433a83990d05d7c79409 (table rebalancer_dynamic_table_new_name [id=cb9d694cb18d4ba6960c1bf712c42e40]) successfully deleted
I20240502 21:27:15.252303 8744 ts_tablet_manager.cc:1909] T 19c14a92c89e433a83990d05d7c79409 P 39981a8dfe94462b9a73ba489af4c714: Deleting consensus metadata
I20240502 21:27:15.252522 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet 19c14a92c89e433a83990d05d7c79409 (table rebalancer_dynamic_table_new_name [id=cb9d694cb18d4ba6960c1bf712c42e40]) successfully deleted
I20240502 21:27:15.253257 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet 19c14a92c89e433a83990d05d7c79409 (table rebalancer_dynamic_table_new_name [id=cb9d694cb18d4ba6960c1bf712c42e40]) successfully deleted
I20240502 21:27:15.338191 4298 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:15.347601 4422 tablet_service.cc:1467] Processing CreateTablet for tablet e07f0c3141344b9d8eb45b1a06c8a949 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=deaff4a08fa4498894f5488fb701805e]), partition=
I20240502 21:27:15.347908 7194 tablet_service.cc:1467] Processing CreateTablet for tablet e07f0c3141344b9d8eb45b1a06c8a949 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=deaff4a08fa4498894f5488fb701805e]), partition=
I20240502 21:27:15.348073 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e07f0c3141344b9d8eb45b1a06c8a949. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:15.348019 7463 tablet_service.cc:1467] Processing CreateTablet for tablet e07f0c3141344b9d8eb45b1a06c8a949 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=deaff4a08fa4498894f5488fb701805e]), partition=
I20240502 21:27:15.348405 7463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e07f0c3141344b9d8eb45b1a06c8a949. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:15.348408 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e07f0c3141344b9d8eb45b1a06c8a949. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:15.354460 8732 tablet_bootstrap.cc:492] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:15.356159 8732 tablet_bootstrap.cc:654] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:15.357796 8747 tablet_bootstrap.cc:492] T e07f0c3141344b9d8eb45b1a06c8a949 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:15.358040 8732 tablet_bootstrap.cc:492] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:15.358270 8732 ts_tablet_manager.cc:1367] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 21:27:15.359089 8732 raft_consensus.cc:348] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:15.359329 8732 raft_consensus.cc:374] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:15.359455 8732 raft_consensus.cc:729] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:15.359612 8747 tablet_bootstrap.cc:654] T e07f0c3141344b9d8eb45b1a06c8a949 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:15.359788 8732 consensus_queue.cc:260] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:15.360558 8732 ts_tablet_manager.cc:1398] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:15.375486 8727 tablet_bootstrap.cc:492] T e07f0c3141344b9d8eb45b1a06c8a949 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:15.377732 8727 tablet_bootstrap.cc:654] T e07f0c3141344b9d8eb45b1a06c8a949 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:15.380378 8727 tablet_bootstrap.cc:492] T e07f0c3141344b9d8eb45b1a06c8a949 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:15.380606 8727 ts_tablet_manager.cc:1367] T e07f0c3141344b9d8eb45b1a06c8a949 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240502 21:27:15.381779 8727 raft_consensus.cc:348] T e07f0c3141344b9d8eb45b1a06c8a949 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:15.382140 8727 raft_consensus.cc:374] T e07f0c3141344b9d8eb45b1a06c8a949 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:15.382241 8727 raft_consensus.cc:729] T e07f0c3141344b9d8eb45b1a06c8a949 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:15.382570 8727 consensus_queue.cc:260] T e07f0c3141344b9d8eb45b1a06c8a949 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:15.384434 8727 ts_tablet_manager.cc:1398] T e07f0c3141344b9d8eb45b1a06c8a949 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240502 21:27:15.388753 8747 tablet_bootstrap.cc:492] T e07f0c3141344b9d8eb45b1a06c8a949 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:15.388909 8747 ts_tablet_manager.cc:1367] T e07f0c3141344b9d8eb45b1a06c8a949 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.031s user 0.016s sys 0.013s
I20240502 21:27:15.389760 8747 raft_consensus.cc:348] T e07f0c3141344b9d8eb45b1a06c8a949 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:15.389997 8747 raft_consensus.cc:374] T e07f0c3141344b9d8eb45b1a06c8a949 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:15.390101 8747 raft_consensus.cc:729] T e07f0c3141344b9d8eb45b1a06c8a949 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:15.390386 8747 consensus_queue.cc:260] T e07f0c3141344b9d8eb45b1a06c8a949 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:15.390992 8747 ts_tablet_manager.cc:1398] T e07f0c3141344b9d8eb45b1a06c8a949 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:15.405978 8654 raft_consensus.cc:482] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:15.406232 8654 raft_consensus.cc:504] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:15.406967 8654 leader_election.cc:288] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:15.407379 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e07f0c3141344b9d8eb45b1a06c8a949" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:15.407471 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e07f0c3141344b9d8eb45b1a06c8a949" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:15.407680 7214 raft_consensus.cc:2454] T e07f0c3141344b9d8eb45b1a06c8a949 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:15.407720 7484 raft_consensus.cc:2454] T e07f0c3141344b9d8eb45b1a06c8a949 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:15.408159 4377 leader_election.cc:302] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:15.408452 8654 raft_consensus.cc:2781] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:15.408561 8654 raft_consensus.cc:482] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:15.408663 8654 raft_consensus.cc:3037] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:15.410727 8654 raft_consensus.cc:504] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:15.411556 8654 leader_election.cc:288] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:15.411926 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e07f0c3141344b9d8eb45b1a06c8a949" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:15.412158 7214 raft_consensus.cc:3037] T e07f0c3141344b9d8eb45b1a06c8a949 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:15.412212 7482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e07f0c3141344b9d8eb45b1a06c8a949" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:15.412406 7482 raft_consensus.cc:3037] T e07f0c3141344b9d8eb45b1a06c8a949 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:15.414991 7482 raft_consensus.cc:2454] T e07f0c3141344b9d8eb45b1a06c8a949 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:15.416373 4375 leader_election.cc:302] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:15.416707 8654 raft_consensus.cc:2781] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:15.417210 8654 raft_consensus.cc:686] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:27:15.417541 8654 consensus_queue.cc:237] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:15.418769 7214 raft_consensus.cc:2454] T e07f0c3141344b9d8eb45b1a06c8a949 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:15.421562 4298 catalog_manager.cc:5484] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:27:15.462314 8661 raft_consensus.cc:482] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:15.462495 8661 raft_consensus.cc:504] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:15.463150 8661 leader_election.cc:288] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:15.463654 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ffa252784984df1bbb0ce35a26d4865" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:15.463907 7618 raft_consensus.cc:2454] T 8ffa252784984df1bbb0ce35a26d4865 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:27:15.464305 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ffa252784984df1bbb0ce35a26d4865" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:15.464399 4509 leader_election.cc:302] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:27:15.464545 7349 raft_consensus.cc:2454] T 8ffa252784984df1bbb0ce35a26d4865 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:27:15.464737 8425 raft_consensus.cc:2781] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:15.464846 8425 raft_consensus.cc:482] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:15.464954 8425 raft_consensus.cc:3037] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:15.467265 8425 raft_consensus.cc:504] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:15.467995 8425 leader_election.cc:288] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Requested vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:15.468255 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ffa252784984df1bbb0ce35a26d4865" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:15.468489 7618 raft_consensus.cc:3037] T 8ffa252784984df1bbb0ce35a26d4865 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:15.468619 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ffa252784984df1bbb0ce35a26d4865" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:15.468823 7350 raft_consensus.cc:3037] T 8ffa252784984df1bbb0ce35a26d4865 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:15.470773 7618 raft_consensus.cc:2454] T 8ffa252784984df1bbb0ce35a26d4865 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:27:15.471132 4509 leader_election.cc:302] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:27:15.471271 7350 raft_consensus.cc:2454] T 8ffa252784984df1bbb0ce35a26d4865 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:27:15.471443 8425 raft_consensus.cc:2781] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:15.471635 8425 raft_consensus.cc:686] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:27:15.471971 8425 consensus_queue.cc:237] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:15.475036 4298 catalog_manager.cc:5484] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:15.531847 4298 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:15.538427 7482 raft_consensus.cc:1261] T e07f0c3141344b9d8eb45b1a06c8a949 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:15.538981 7214 raft_consensus.cc:1261] T e07f0c3141344b9d8eb45b1a06c8a949 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:15.539057 8654 consensus_queue.cc:1035] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:15.539790 8654 consensus_queue.cc:1035] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:15.544871 8757 tablet.cc:1722] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:15.545590 8757 tablet.cc:1620] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:15.546617 8758 tablet.cc:1722] T e07f0c3141344b9d8eb45b1a06c8a949 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:15.547235 4298 catalog_manager.cc:5576] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913) does not have the latest schema for tablet e07f0c3141344b9d8eb45b1a06c8a949 (table rebalancer_dynamic_table [id=deaff4a08fa4498894f5488fb701805e]). Expected version 1 got 0
I20240502 21:27:15.547386 8758 tablet.cc:1620] T e07f0c3141344b9d8eb45b1a06c8a949 P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:15.555619 4291 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=deaff4a08fa4498894f5488fb701805e] alter complete (version 1)
I20240502 21:27:15.560863 8715 tablet.cc:1722] T e07f0c3141344b9d8eb45b1a06c8a949 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:15.561307 8715 tablet.cc:1620] T e07f0c3141344b9d8eb45b1a06c8a949 P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:15.561525 4291 catalog_manager.cc:5576] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193) does not have the latest schema for tablet e07f0c3141344b9d8eb45b1a06c8a949 (table rebalancer_dynamic_table [id=deaff4a08fa4498894f5488fb701805e]). Expected version 1 got 0
I20240502 21:27:15.580209 7942 raft_consensus.cc:482] T a8a3d57ff2184ab3bc09140afbffe0b8 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:15.580415 7942 raft_consensus.cc:504] T a8a3d57ff2184ab3bc09140afbffe0b8 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:15.581224 7942 leader_election.cc:288] T a8a3d57ff2184ab3bc09140afbffe0b8 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:27:15.581686 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a3d57ff2184ab3bc09140afbffe0b8" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:15.581929 7214 raft_consensus.cc:2454] T a8a3d57ff2184ab3bc09140afbffe0b8 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:15.581964 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a3d57ff2184ab3bc09140afbffe0b8" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:27:15.582211 4706 raft_consensus.cc:2454] T a8a3d57ff2184ab3bc09140afbffe0b8 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:15.582449 7551 leader_election.cc:302] T a8a3d57ff2184ab3bc09140afbffe0b8 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:15.582795 7942 raft_consensus.cc:2781] T a8a3d57ff2184ab3bc09140afbffe0b8 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:15.582883 7942 raft_consensus.cc:482] T a8a3d57ff2184ab3bc09140afbffe0b8 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:15.582983 7942 raft_consensus.cc:3037] T a8a3d57ff2184ab3bc09140afbffe0b8 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:15.584923 7942 raft_consensus.cc:504] T a8a3d57ff2184ab3bc09140afbffe0b8 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:15.585541 7942 leader_election.cc:288] T a8a3d57ff2184ab3bc09140afbffe0b8 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Requested vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:27:15.585978 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a3d57ff2184ab3bc09140afbffe0b8" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:15.586091 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a3d57ff2184ab3bc09140afbffe0b8" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:27:15.586236 7214 raft_consensus.cc:3037] T a8a3d57ff2184ab3bc09140afbffe0b8 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:15.586318 4706 raft_consensus.cc:3037] T a8a3d57ff2184ab3bc09140afbffe0b8 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:15.588536 4706 raft_consensus.cc:2454] T a8a3d57ff2184ab3bc09140afbffe0b8 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:15.588609 7214 raft_consensus.cc:2454] T a8a3d57ff2184ab3bc09140afbffe0b8 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:15.588943 7552 leader_election.cc:302] T a8a3d57ff2184ab3bc09140afbffe0b8 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 398c3c8f3195492c900439ad5e1931bb; no voters:
I20240502 21:27:15.589217 7942 raft_consensus.cc:2781] T a8a3d57ff2184ab3bc09140afbffe0b8 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:15.589414 7942 raft_consensus.cc:686] T a8a3d57ff2184ab3bc09140afbffe0b8 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Becoming Leader. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Running, Role: LEADER
I20240502 21:27:15.589713 7942 consensus_queue.cc:237] T a8a3d57ff2184ab3bc09140afbffe0b8 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:15.593322 4291 catalog_manager.cc:5484] T a8a3d57ff2184ab3bc09140afbffe0b8 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ddd486109f5434cace13d60d633ac04 (127.29.82.72). New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:15.616699 7942 consensus_queue.cc:1035] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:15.623194 7942 consensus_queue.cc:1035] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:15.657610 4291 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:15.665012 8757 tablet.cc:1722] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:15.665545 8757 tablet.cc:1620] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:15.666559 8758 tablet.cc:1722] T e07f0c3141344b9d8eb45b1a06c8a949 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:15.667029 8758 tablet.cc:1620] T e07f0c3141344b9d8eb45b1a06c8a949 P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:15.674038 8715 tablet.cc:1722] T e07f0c3141344b9d8eb45b1a06c8a949 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:15.674554 8715 tablet.cc:1620] T e07f0c3141344b9d8eb45b1a06c8a949 P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:15.675499 4294 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=deaff4a08fa4498894f5488fb701805e] alter complete (version 2)
I20240502 21:27:15.777890 4298 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:15.786724 8757 tablet.cc:1722] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:15.788319 8758 tablet.cc:1722] T e07f0c3141344b9d8eb45b1a06c8a949 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:15.794337 8715 tablet.cc:1722] T e07f0c3141344b9d8eb45b1a06c8a949 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:15.796650 4298 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=deaff4a08fa4498894f5488fb701805e] alter complete (version 3)
I20240502 21:27:15.882849 8425 consensus_queue.cc:1035] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:15.888165 8661 consensus_queue.cc:1035] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:15.901304 4298 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:15.901496 4298 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:15.904151 4298 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet e07f0c3141344b9d8eb45b1a06c8a949
I20240502 21:27:15.904990 4422 tablet_service.cc:1514] Processing DeleteTablet for tablet e07f0c3141344b9d8eb45b1a06c8a949 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:15 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:15.905006 7194 tablet_service.cc:1514] Processing DeleteTablet for tablet e07f0c3141344b9d8eb45b1a06c8a949 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:15 UTC) from {username='slave'} at 127.0.0.1:44652
I20240502 21:27:15.905395 7463 tablet_service.cc:1514] Processing DeleteTablet for tablet e07f0c3141344b9d8eb45b1a06c8a949 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:15 UTC) from {username='slave'} at 127.0.0.1:37182
I20240502 21:27:15.905737 8772 tablet_replica.cc:330] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:15.906080 8772 raft_consensus.cc:2229] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:15.906585 8772 raft_consensus.cc:2258] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:15.907073 8774 tablet_replica.cc:330] T e07f0c3141344b9d8eb45b1a06c8a949 P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:27:15.907457 8774 raft_consensus.cc:2229] T e07f0c3141344b9d8eb45b1a06c8a949 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:15.908176 8772 ts_tablet_manager.cc:1875] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:15.908294 8774 raft_consensus.cc:2258] T e07f0c3141344b9d8eb45b1a06c8a949 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:15.909829 8774 ts_tablet_manager.cc:1875] T e07f0c3141344b9d8eb45b1a06c8a949 P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:15.914626 8772 ts_tablet_manager.cc:1888] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:15.914767 8772 log.cc:1196] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/e07f0c3141344b9d8eb45b1a06c8a949
I20240502 21:27:15.915230 8772 ts_tablet_manager.cc:1909] T e07f0c3141344b9d8eb45b1a06c8a949 P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:15.916071 8773 tablet_replica.cc:330] T e07f0c3141344b9d8eb45b1a06c8a949 P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:27:15.916343 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet e07f0c3141344b9d8eb45b1a06c8a949 (table rebalancer_dynamic_table_new_name [id=deaff4a08fa4498894f5488fb701805e]) successfully deleted
I20240502 21:27:15.916594 8773 raft_consensus.cc:2229] T e07f0c3141344b9d8eb45b1a06c8a949 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:15.916671 8774 ts_tablet_manager.cc:1888] T e07f0c3141344b9d8eb45b1a06c8a949 P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:15.916786 8774 log.cc:1196] T e07f0c3141344b9d8eb45b1a06c8a949 P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/e07f0c3141344b9d8eb45b1a06c8a949
I20240502 21:27:15.917289 8774 ts_tablet_manager.cc:1909] T e07f0c3141344b9d8eb45b1a06c8a949 P 31a72286397c4edc9a63eca3412be168: Deleting consensus metadata
I20240502 21:27:15.917317 8773 raft_consensus.cc:2258] T e07f0c3141344b9d8eb45b1a06c8a949 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:15.918249 8773 ts_tablet_manager.cc:1875] T e07f0c3141344b9d8eb45b1a06c8a949 P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:15.919960 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet e07f0c3141344b9d8eb45b1a06c8a949 (table rebalancer_dynamic_table_new_name [id=deaff4a08fa4498894f5488fb701805e]) successfully deleted
I20240502 21:27:15.924427 8773 ts_tablet_manager.cc:1888] T e07f0c3141344b9d8eb45b1a06c8a949 P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:15.924546 8773 log.cc:1196] T e07f0c3141344b9d8eb45b1a06c8a949 P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/e07f0c3141344b9d8eb45b1a06c8a949
I20240502 21:27:15.924945 8773 ts_tablet_manager.cc:1909] T e07f0c3141344b9d8eb45b1a06c8a949 P 39981a8dfe94462b9a73ba489af4c714: Deleting consensus metadata
I20240502 21:27:15.925992 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet e07f0c3141344b9d8eb45b1a06c8a949 (table rebalancer_dynamic_table_new_name [id=deaff4a08fa4498894f5488fb701805e]) successfully deleted
I20240502 21:27:15.967736 7942 consensus_queue.cc:1035] T a8a3d57ff2184ab3bc09140afbffe0b8 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:15.972091 7942 consensus_queue.cc:1035] T a8a3d57ff2184ab3bc09140afbffe0b8 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:16.008783 4291 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:16.016804 7329 tablet_service.cc:1467] Processing CreateTablet for tablet 854d75f5c1574249b8726a00e0189dc4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=72456c0620c64c499b490c020bc04a27]), partition=
I20240502 21:27:16.016836 7194 tablet_service.cc:1467] Processing CreateTablet for tablet 854d75f5c1574249b8726a00e0189dc4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=72456c0620c64c499b490c020bc04a27]), partition=
I20240502 21:27:16.017215 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 854d75f5c1574249b8726a00e0189dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:16.017222 7329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 854d75f5c1574249b8726a00e0189dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:16.017243 7597 tablet_service.cc:1467] Processing CreateTablet for tablet 854d75f5c1574249b8726a00e0189dc4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=72456c0620c64c499b490c020bc04a27]), partition=
I20240502 21:27:16.017597 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 854d75f5c1574249b8726a00e0189dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:16.028918 8778 tablet_bootstrap.cc:492] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:16.029879 8779 tablet_bootstrap.cc:492] T 854d75f5c1574249b8726a00e0189dc4 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:16.032178 8778 tablet_bootstrap.cc:654] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:16.032497 8779 tablet_bootstrap.cc:654] T 854d75f5c1574249b8726a00e0189dc4 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:16.036873 8779 tablet_bootstrap.cc:492] T 854d75f5c1574249b8726a00e0189dc4 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:16.037871 8779 ts_tablet_manager.cc:1367] T 854d75f5c1574249b8726a00e0189dc4 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240502 21:27:16.038995 8779 raft_consensus.cc:348] T 854d75f5c1574249b8726a00e0189dc4 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:16.039214 8779 raft_consensus.cc:374] T 854d75f5c1574249b8726a00e0189dc4 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:16.039326 8779 raft_consensus.cc:729] T 854d75f5c1574249b8726a00e0189dc4 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:16.039623 8779 consensus_queue.cc:260] T 854d75f5c1574249b8726a00e0189dc4 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:16.041049 8778 tablet_bootstrap.cc:492] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:16.041229 8778 ts_tablet_manager.cc:1367] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.006s
I20240502 21:27:16.041468 8780 tablet_bootstrap.cc:492] T 854d75f5c1574249b8726a00e0189dc4 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:16.041903 8779 ts_tablet_manager.cc:1398] T 854d75f5c1574249b8726a00e0189dc4 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:16.042043 8778 raft_consensus.cc:348] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:16.042301 8778 raft_consensus.cc:374] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:16.042398 8778 raft_consensus.cc:729] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:16.042677 8778 consensus_queue.cc:260] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:16.043227 8780 tablet_bootstrap.cc:654] T 854d75f5c1574249b8726a00e0189dc4 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:16.043444 8778 ts_tablet_manager.cc:1398] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 21:27:16.045773 8780 tablet_bootstrap.cc:492] T 854d75f5c1574249b8726a00e0189dc4 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:16.047943 8780 ts_tablet_manager.cc:1367] T 854d75f5c1574249b8726a00e0189dc4 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 21:27:16.048625 8780 raft_consensus.cc:348] T 854d75f5c1574249b8726a00e0189dc4 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:16.048907 8780 raft_consensus.cc:374] T 854d75f5c1574249b8726a00e0189dc4 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:16.049000 8780 raft_consensus.cc:729] T 854d75f5c1574249b8726a00e0189dc4 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:16.049234 8780 consensus_queue.cc:260] T 854d75f5c1574249b8726a00e0189dc4 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:16.049703 8780 ts_tablet_manager.cc:1398] T 854d75f5c1574249b8726a00e0189dc4 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:16.096303 8651 raft_consensus.cc:482] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:16.096503 8651 raft_consensus.cc:504] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:16.097329 8651 leader_election.cc:288] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:16.097784 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "854d75f5c1574249b8726a00e0189dc4" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:16.097903 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "854d75f5c1574249b8726a00e0189dc4" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:16.098071 7350 raft_consensus.cc:2454] T 854d75f5c1574249b8726a00e0189dc4 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:16.098137 7618 raft_consensus.cc:2454] T 854d75f5c1574249b8726a00e0189dc4 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:16.098503 7150 leader_election.cc:302] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:16.098789 8651 raft_consensus.cc:2781] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:16.098889 8651 raft_consensus.cc:482] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:16.098955 8651 raft_consensus.cc:3037] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:16.100875 8651 raft_consensus.cc:504] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:16.101387 8651 leader_election.cc:288] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:16.101845 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "854d75f5c1574249b8726a00e0189dc4" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:16.101838 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "854d75f5c1574249b8726a00e0189dc4" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:16.102018 7618 raft_consensus.cc:3037] T 854d75f5c1574249b8726a00e0189dc4 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:16.102054 7350 raft_consensus.cc:3037] T 854d75f5c1574249b8726a00e0189dc4 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:16.104108 7618 raft_consensus.cc:2454] T 854d75f5c1574249b8726a00e0189dc4 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:16.104121 7350 raft_consensus.cc:2454] T 854d75f5c1574249b8726a00e0189dc4 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:16.104503 7149 leader_election.cc:302] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:16.104883 8651 raft_consensus.cc:2781] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:16.105051 8651 raft_consensus.cc:686] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:27:16.105396 8651 consensus_queue.cc:237] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:16.108662 4298 catalog_manager.cc:5484] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69). New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:16.160444 4298 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:16.169286 4555 tablet_service.cc:1467] Processing CreateTablet for tablet 4c06f3bc97254bff8181d3f6f02470b4 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=569b30a95e114b50ada524f1380f198e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:16.169400 7194 tablet_service.cc:1467] Processing CreateTablet for tablet 4c06f3bc97254bff8181d3f6f02470b4 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=569b30a95e114b50ada524f1380f198e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:16.169878 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c06f3bc97254bff8181d3f6f02470b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:16.169874 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c06f3bc97254bff8181d3f6f02470b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:16.170190 7329 tablet_service.cc:1467] Processing CreateTablet for tablet be423032bc4b459b8e8e63a2379e37ac (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=569b30a95e114b50ada524f1380f198e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:16.170346 7195 tablet_service.cc:1467] Processing CreateTablet for tablet be423032bc4b459b8e8e63a2379e37ac (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=569b30a95e114b50ada524f1380f198e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:16.170631 7329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be423032bc4b459b8e8e63a2379e37ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:16.170746 7195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be423032bc4b459b8e8e63a2379e37ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:16.177291 8780 tablet_bootstrap.cc:492] T be423032bc4b459b8e8e63a2379e37ac P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:16.177824 7597 tablet_service.cc:1467] Processing CreateTablet for tablet 4c06f3bc97254bff8181d3f6f02470b4 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=569b30a95e114b50ada524f1380f198e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:16.178325 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c06f3bc97254bff8181d3f6f02470b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:16.178997 8780 tablet_bootstrap.cc:654] T be423032bc4b459b8e8e63a2379e37ac P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:16.180284 7595 tablet_service.cc:1467] Processing CreateTablet for tablet be423032bc4b459b8e8e63a2379e37ac (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=569b30a95e114b50ada524f1380f198e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:16.180692 7595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be423032bc4b459b8e8e63a2379e37ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:16.181254 8787 tablet_bootstrap.cc:492] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:16.182499 8787 tablet_bootstrap.cc:654] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:16.186530 7328 tablet_service.cc:1467] Processing CreateTablet for tablet a410b7da18004b23bdad650f59a30e8c (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=569b30a95e114b50ada524f1380f198e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:16.187021 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a410b7da18004b23bdad650f59a30e8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:16.187886 7194 tablet_service.cc:1467] Processing CreateTablet for tablet a410b7da18004b23bdad650f59a30e8c (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=569b30a95e114b50ada524f1380f198e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:16.188304 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a410b7da18004b23bdad650f59a30e8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:16.190766 4686 tablet_service.cc:1467] Processing CreateTablet for tablet a410b7da18004b23bdad650f59a30e8c (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=569b30a95e114b50ada524f1380f198e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:16.191179 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a410b7da18004b23bdad650f59a30e8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:16.195932 8778 tablet_bootstrap.cc:492] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:16.195919 8779 tablet_bootstrap.cc:492] T 4c06f3bc97254bff8181d3f6f02470b4 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:16.197994 8778 tablet_bootstrap.cc:654] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:16.198107 8779 tablet_bootstrap.cc:654] T 4c06f3bc97254bff8181d3f6f02470b4 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:16.202930 8778 tablet_bootstrap.cc:492] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:16.203119 8778 ts_tablet_manager.cc:1367] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240502 21:27:16.204022 8778 raft_consensus.cc:348] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:16.204257 8778 raft_consensus.cc:374] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:16.204339 8778 raft_consensus.cc:729] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:16.204609 8778 consensus_queue.cc:260] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:16.205195 8778 ts_tablet_manager.cc:1398] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:16.205533 8778 tablet_bootstrap.cc:492] T 4c06f3bc97254bff8181d3f6f02470b4 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:16.207096 8778 tablet_bootstrap.cc:654] T 4c06f3bc97254bff8181d3f6f02470b4 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:16.209460 8787 tablet_bootstrap.cc:492] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:16.209702 8787 ts_tablet_manager.cc:1367] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.029s user 0.015s sys 0.012s
I20240502 21:27:16.210981 8787 raft_consensus.cc:348] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:16.211398 8787 raft_consensus.cc:374] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:16.211542 8787 raft_consensus.cc:729] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:16.211867 8787 consensus_queue.cc:260] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:16.212580 8787 ts_tablet_manager.cc:1398] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:16.218559 8780 tablet_bootstrap.cc:492] T be423032bc4b459b8e8e63a2379e37ac P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:16.218835 8780 ts_tablet_manager.cc:1367] T be423032bc4b459b8e8e63a2379e37ac P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.042s user 0.000s sys 0.003s
I20240502 21:27:16.219869 8780 raft_consensus.cc:348] T be423032bc4b459b8e8e63a2379e37ac P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:16.225598 8779 tablet_bootstrap.cc:492] T 4c06f3bc97254bff8181d3f6f02470b4 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:16.225883 8779 ts_tablet_manager.cc:1367] T 4c06f3bc97254bff8181d3f6f02470b4 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.000s
I20240502 21:27:16.226338 8780 raft_consensus.cc:374] T be423032bc4b459b8e8e63a2379e37ac P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:16.226476 8780 raft_consensus.cc:729] T be423032bc4b459b8e8e63a2379e37ac P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:16.226783 8780 consensus_queue.cc:260] T be423032bc4b459b8e8e63a2379e37ac P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:16.226576 8793 tablet_bootstrap.cc:492] T a410b7da18004b23bdad650f59a30e8c P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:16.229202 8780 ts_tablet_manager.cc:1398] T be423032bc4b459b8e8e63a2379e37ac P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240502 21:27:16.229569 8780 tablet_bootstrap.cc:492] T a410b7da18004b23bdad650f59a30e8c P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:16.230053 8793 tablet_bootstrap.cc:654] T a410b7da18004b23bdad650f59a30e8c P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:16.231338 4291 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:16.231782 8780 tablet_bootstrap.cc:654] T a410b7da18004b23bdad650f59a30e8c P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:16.234043 8778 tablet_bootstrap.cc:492] T 4c06f3bc97254bff8181d3f6f02470b4 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:16.234335 8778 ts_tablet_manager.cc:1367] T 4c06f3bc97254bff8181d3f6f02470b4 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.012s
W20240502 21:27:16.235141 4639 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.68:42449, user_credentials={real_user=slave}} blocked reactor thread for 49271.3us
I20240502 21:27:16.238504 8793 tablet_bootstrap.cc:492] T a410b7da18004b23bdad650f59a30e8c P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:27:16.238760 8793 ts_tablet_manager.cc:1367] T a410b7da18004b23bdad650f59a30e8c P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20240502 21:27:16.239712 8793 raft_consensus.cc:348] T a410b7da18004b23bdad650f59a30e8c P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:16.239962 7618 raft_consensus.cc:1261] T 854d75f5c1574249b8726a00e0189dc4 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:16.239989 8793 raft_consensus.cc:374] T a410b7da18004b23bdad650f59a30e8c P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:16.240166 8793 raft_consensus.cc:729] T a410b7da18004b23bdad650f59a30e8c P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:16.240181 7350 raft_consensus.cc:1261] T 854d75f5c1574249b8726a00e0189dc4 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:16.240442 8793 consensus_queue.cc:260] T a410b7da18004b23bdad650f59a30e8c P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:16.240567 8649 consensus_queue.cc:1035] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:16.241035 8649 consensus_queue.cc:1035] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:16.241164 8793 ts_tablet_manager.cc:1398] T a410b7da18004b23bdad650f59a30e8c P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:16.244154 8778 raft_consensus.cc:348] T 4c06f3bc97254bff8181d3f6f02470b4 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:16.244643 8778 raft_consensus.cc:374] T 4c06f3bc97254bff8181d3f6f02470b4 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:16.244817 8778 raft_consensus.cc:729] T 4c06f3bc97254bff8181d3f6f02470b4 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:16.245532 8778 consensus_queue.cc:260] T 4c06f3bc97254bff8181d3f6f02470b4 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:16.252142 8778 ts_tablet_manager.cc:1398] T 4c06f3bc97254bff8181d3f6f02470b4 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20240502 21:27:16.253901 8778 tablet_bootstrap.cc:492] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:16.255733 8778 tablet_bootstrap.cc:654] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:16.261341 8715 tablet.cc:1722] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:16.261948 8715 tablet.cc:1620] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:16.264823 8778 tablet_bootstrap.cc:492] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:16.265025 8778 ts_tablet_manager.cc:1367] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.009s
I20240502 21:27:16.265866 4300 catalog_manager.cc:5576] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223) does not have the latest schema for tablet 854d75f5c1574249b8726a00e0189dc4 (table rebalancer_dynamic_table [id=72456c0620c64c499b490c020bc04a27]). Expected version 1 got 0
I20240502 21:27:16.266145 8778 raft_consensus.cc:348] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:16.266441 8778 raft_consensus.cc:374] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:16.266541 8778 raft_consensus.cc:729] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:16.266883 8778 consensus_queue.cc:260] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:16.267764 8778 ts_tablet_manager.cc:1398] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:27:16.273321 8800 tablet.cc:1722] T 854d75f5c1574249b8726a00e0189dc4 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:16.275609 8800 tablet.cc:1620] T 854d75f5c1574249b8726a00e0189dc4 P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:16.280165 8780 tablet_bootstrap.cc:492] T a410b7da18004b23bdad650f59a30e8c P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:16.280557 4292 catalog_manager.cc:5576] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719) does not have the latest schema for tablet 854d75f5c1574249b8726a00e0189dc4 (table rebalancer_dynamic_table [id=72456c0620c64c499b490c020bc04a27]). Expected version 1 got 0
I20240502 21:27:16.280645 8780 ts_tablet_manager.cc:1367] T a410b7da18004b23bdad650f59a30e8c P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.051s user 0.001s sys 0.006s
I20240502 21:27:16.281680 8780 raft_consensus.cc:348] T a410b7da18004b23bdad650f59a30e8c P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:16.282173 8780 raft_consensus.cc:374] T a410b7da18004b23bdad650f59a30e8c P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:16.282357 8780 raft_consensus.cc:729] T a410b7da18004b23bdad650f59a30e8c P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:16.282760 8780 consensus_queue.cc:260] T a410b7da18004b23bdad650f59a30e8c P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:16.283526 8780 ts_tablet_manager.cc:1398] T a410b7da18004b23bdad650f59a30e8c P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:27:16.290277 8801 tablet.cc:1722] T 854d75f5c1574249b8726a00e0189dc4 P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:16.291445 8801 tablet.cc:1620] T 854d75f5c1574249b8726a00e0189dc4 P 0ddd486109f5434cace13d60d633ac04: MemRowSet was empty: no flush needed.
I20240502 21:27:16.292850 4292 catalog_manager.cc:5576] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223) does not have the latest schema for tablet 854d75f5c1574249b8726a00e0189dc4 (table rebalancer_dynamic_table [id=72456c0620c64c499b490c020bc04a27]). Expected version 1 got 0
I20240502 21:27:16.298647 4294 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=72456c0620c64c499b490c020bc04a27] alter complete (version 1)
I20240502 21:27:16.226981 8779 raft_consensus.cc:348] T 4c06f3bc97254bff8181d3f6f02470b4 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:16.300041 8779 raft_consensus.cc:374] T 4c06f3bc97254bff8181d3f6f02470b4 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:16.300246 8779 raft_consensus.cc:729] T 4c06f3bc97254bff8181d3f6f02470b4 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:16.300609 8779 consensus_queue.cc:260] T 4c06f3bc97254bff8181d3f6f02470b4 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:16.301240 8779 ts_tablet_manager.cc:1398] T 4c06f3bc97254bff8181d3f6f02470b4 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.075s user 0.002s sys 0.000s
I20240502 21:27:16.301800 8779 tablet_bootstrap.cc:492] T be423032bc4b459b8e8e63a2379e37ac P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:16.307338 8779 tablet_bootstrap.cc:654] T be423032bc4b459b8e8e63a2379e37ac P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:16.309065 8661 raft_consensus.cc:482] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:16.309252 8661 raft_consensus.cc:504] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:16.310030 8661 leader_election.cc:288] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:16.310847 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c06f3bc97254bff8181d3f6f02470b4" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:16.311121 7215 raft_consensus.cc:2454] T 4c06f3bc97254bff8181d3f6f02470b4 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:27:16.312078 4510 leader_election.cc:302] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:27:16.312536 8661 raft_consensus.cc:2781] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:16.312625 8661 raft_consensus.cc:482] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:16.312738 8661 raft_consensus.cc:3037] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:16.315564 8661 raft_consensus.cc:504] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:16.317384 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c06f3bc97254bff8181d3f6f02470b4" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:16.317629 7215 raft_consensus.cc:3037] T 4c06f3bc97254bff8181d3f6f02470b4 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:16.326328 7942 raft_consensus.cc:482] T 4c06f3bc97254bff8181d3f6f02470b4 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:16.326525 7942 raft_consensus.cc:504] T 4c06f3bc97254bff8181d3f6f02470b4 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:16.327445 7942 leader_election.cc:288] T 4c06f3bc97254bff8181d3f6f02470b4 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:16.327983 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c06f3bc97254bff8181d3f6f02470b4" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:27:16.328308 4575 raft_consensus.cc:2379] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ddd486109f5434cace13d60d633ac04 in current term 1: Already voted for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in this term.
I20240502 21:27:16.328377 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c06f3bc97254bff8181d3f6f02470b4" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:16.329316 7552 leader_election.cc:302] T 4c06f3bc97254bff8181d3f6f02470b4 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04; no voters: 39981a8dfe94462b9a73ba489af4c714, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:27:16.329639 7942 raft_consensus.cc:3037] T 4c06f3bc97254bff8181d3f6f02470b4 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:16.332365 7942 raft_consensus.cc:2726] T 4c06f3bc97254bff8181d3f6f02470b4 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:27:16.333942 7215 raft_consensus.cc:2454] T 4c06f3bc97254bff8181d3f6f02470b4 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:27:16.335906 4510 leader_election.cc:302] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:27:16.336232 8425 raft_consensus.cc:2781] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:16.336410 8425 raft_consensus.cc:686] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:27:16.336666 8425 consensus_queue.cc:237] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:16.335675 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c06f3bc97254bff8181d3f6f02470b4" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:16.339046 7617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c06f3bc97254bff8181d3f6f02470b4" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:16.339936 8661 leader_election.cc:288] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Requested vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:16.341059 8779 tablet_bootstrap.cc:492] T be423032bc4b459b8e8e63a2379e37ac P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:16.341253 8779 ts_tablet_manager.cc:1367] T be423032bc4b459b8e8e63a2379e37ac P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
I20240502 21:27:16.342084 7617 raft_consensus.cc:2454] T 4c06f3bc97254bff8181d3f6f02470b4 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:27:16.343345 7618 raft_consensus.cc:2362] T 4c06f3bc97254bff8181d3f6f02470b4 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1. Re-sending same reply.
I20240502 21:27:16.342293 8779 raft_consensus.cc:348] T be423032bc4b459b8e8e63a2379e37ac P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:16.344184 8779 raft_consensus.cc:374] T be423032bc4b459b8e8e63a2379e37ac P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:16.344339 8779 raft_consensus.cc:729] T be423032bc4b459b8e8e63a2379e37ac P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:16.344612 4294 catalog_manager.cc:5484] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:16.347249 8779 consensus_queue.cc:260] T be423032bc4b459b8e8e63a2379e37ac P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:16.348695 8779 ts_tablet_manager.cc:1398] T be423032bc4b459b8e8e63a2379e37ac P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 21:27:16.406239 4298 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:16.415766 8715 tablet.cc:1722] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:16.416327 8715 tablet.cc:1620] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:16.417922 8800 tablet.cc:1722] T 854d75f5c1574249b8726a00e0189dc4 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:16.418329 8800 tablet.cc:1620] T 854d75f5c1574249b8726a00e0189dc4 P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:16.419731 8801 tablet.cc:1722] T 854d75f5c1574249b8726a00e0189dc4 P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:16.420217 8801 tablet.cc:1620] T 854d75f5c1574249b8726a00e0189dc4 P 0ddd486109f5434cace13d60d633ac04: MemRowSet was empty: no flush needed.
I20240502 21:27:16.432646 4300 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=72456c0620c64c499b490c020bc04a27] alter complete (version 2)
I20240502 21:27:16.535015 4292 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:16.549393 8715 tablet.cc:1722] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:16.551075 8801 tablet.cc:1722] T 854d75f5c1574249b8726a00e0189dc4 P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:16.552088 8800 tablet.cc:1722] T 854d75f5c1574249b8726a00e0189dc4 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:16.558435 4294 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=72456c0620c64c499b490c020bc04a27] alter complete (version 3)
I20240502 21:27:16.620486 8651 raft_consensus.cc:482] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:16.620828 8651 raft_consensus.cc:504] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:16.621987 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a410b7da18004b23bdad650f59a30e8c" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:16.622361 7350 raft_consensus.cc:2454] T a410b7da18004b23bdad650f59a30e8c P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:16.622097 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a410b7da18004b23bdad650f59a30e8c" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:27:16.622815 4706 raft_consensus.cc:2454] T a410b7da18004b23bdad650f59a30e8c P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:16.623211 8651 leader_election.cc:288] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:27:16.623347 7150 leader_election.cc:302] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:16.623714 8651 raft_consensus.cc:2781] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:16.623834 8651 raft_consensus.cc:482] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:16.623925 8651 raft_consensus.cc:3037] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:16.625852 8651 raft_consensus.cc:504] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:16.626391 8651 leader_election.cc:288] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:27:16.626835 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a410b7da18004b23bdad650f59a30e8c" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:16.626848 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a410b7da18004b23bdad650f59a30e8c" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:27:16.627097 7350 raft_consensus.cc:3037] T a410b7da18004b23bdad650f59a30e8c P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:16.627221 4706 raft_consensus.cc:3037] T a410b7da18004b23bdad650f59a30e8c P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:16.630095 7350 raft_consensus.cc:2454] T a410b7da18004b23bdad650f59a30e8c P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:16.630250 4706 raft_consensus.cc:2454] T a410b7da18004b23bdad650f59a30e8c P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:16.630671 7150 leader_election.cc:302] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:16.630975 8651 raft_consensus.cc:2781] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:16.631137 8651 raft_consensus.cc:686] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:27:16.631393 8651 consensus_queue.cc:237] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:16.635310 4292 catalog_manager.cc:5484] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69). New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:16.652284 8651 raft_consensus.cc:482] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:16.652429 8651 raft_consensus.cc:504] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:16.653012 8651 leader_election.cc:288] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:16.653455 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be423032bc4b459b8e8e63a2379e37ac" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:16.653497 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be423032bc4b459b8e8e63a2379e37ac" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:16.653685 7350 raft_consensus.cc:2454] T be423032bc4b459b8e8e63a2379e37ac P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:16.653687 7618 raft_consensus.cc:2454] T be423032bc4b459b8e8e63a2379e37ac P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:16.654047 7150 leader_election.cc:302] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:16.654289 8651 raft_consensus.cc:2781] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:16.654407 8651 raft_consensus.cc:482] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:16.654502 8651 raft_consensus.cc:3037] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:16.656523 8651 raft_consensus.cc:504] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:16.657127 8651 leader_election.cc:288] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:16.657583 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be423032bc4b459b8e8e63a2379e37ac" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:16.657675 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be423032bc4b459b8e8e63a2379e37ac" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:16.657768 7618 raft_consensus.cc:3037] T be423032bc4b459b8e8e63a2379e37ac P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:16.657873 7350 raft_consensus.cc:3037] T be423032bc4b459b8e8e63a2379e37ac P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:16.659708 7618 raft_consensus.cc:2454] T be423032bc4b459b8e8e63a2379e37ac P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:16.659933 7350 raft_consensus.cc:2454] T be423032bc4b459b8e8e63a2379e37ac P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:16.660087 7149 leader_election.cc:302] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:16.660395 8651 raft_consensus.cc:2781] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:16.660634 8651 raft_consensus.cc:686] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:27:16.660812 4292 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:16.661013 4292 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:16.660964 8651 consensus_queue.cc:237] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:16.664232 4300 catalog_manager.cc:5484] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69). New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:16.664467 4292 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 854d75f5c1574249b8726a00e0189dc4
I20240502 21:27:16.665621 7195 tablet_service.cc:1514] Processing DeleteTablet for tablet 854d75f5c1574249b8726a00e0189dc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:16 UTC) from {username='slave'} at 127.0.0.1:44652
I20240502 21:27:16.665799 7329 tablet_service.cc:1514] Processing DeleteTablet for tablet 854d75f5c1574249b8726a00e0189dc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:16 UTC) from {username='slave'} at 127.0.0.1:47078
I20240502 21:27:16.666152 7595 tablet_service.cc:1514] Processing DeleteTablet for tablet 854d75f5c1574249b8726a00e0189dc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:16 UTC) from {username='slave'} at 127.0.0.1:44444
I20240502 21:27:16.666601 8807 tablet_replica.cc:330] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:27:16.666692 8808 tablet_replica.cc:330] T 854d75f5c1574249b8726a00e0189dc4 P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:27:16.666810 8807 raft_consensus.cc:2229] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:16.666975 8808 raft_consensus.cc:2229] T 854d75f5c1574249b8726a00e0189dc4 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:16.667217 8807 raft_consensus.cc:2258] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:16.667238 8808 raft_consensus.cc:2258] T 854d75f5c1574249b8726a00e0189dc4 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:16.668151 8808 ts_tablet_manager.cc:1875] T 854d75f5c1574249b8726a00e0189dc4 P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:16.668262 8807 ts_tablet_manager.cc:1875] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:16.674613 8807 ts_tablet_manager.cc:1888] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:16.674738 8807 log.cc:1196] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/854d75f5c1574249b8726a00e0189dc4
I20240502 21:27:16.675232 8807 ts_tablet_manager.cc:1909] T 854d75f5c1574249b8726a00e0189dc4 P 39981a8dfe94462b9a73ba489af4c714: Deleting consensus metadata
I20240502 21:27:16.676314 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet 854d75f5c1574249b8726a00e0189dc4 (table rebalancer_dynamic_table_new_name [id=72456c0620c64c499b490c020bc04a27]) successfully deleted
I20240502 21:27:16.677141 8808 ts_tablet_manager.cc:1888] T 854d75f5c1574249b8726a00e0189dc4 P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:16.677292 8808 log.cc:1196] T 854d75f5c1574249b8726a00e0189dc4 P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/854d75f5c1574249b8726a00e0189dc4
I20240502 21:27:16.677778 8808 ts_tablet_manager.cc:1909] T 854d75f5c1574249b8726a00e0189dc4 P d0456c5e55a542128688904e003d403c: Deleting consensus metadata
I20240502 21:27:16.678809 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet 854d75f5c1574249b8726a00e0189dc4 (table rebalancer_dynamic_table_new_name [id=72456c0620c64c499b490c020bc04a27]) successfully deleted
I20240502 21:27:16.679920 8809 tablet_replica.cc:330] T 854d75f5c1574249b8726a00e0189dc4 P 0ddd486109f5434cace13d60d633ac04: stopping tablet replica
I20240502 21:27:16.680153 8809 raft_consensus.cc:2229] T 854d75f5c1574249b8726a00e0189dc4 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:16.680377 8809 raft_consensus.cc:2258] T 854d75f5c1574249b8726a00e0189dc4 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:16.681146 8809 ts_tablet_manager.cc:1875] T 854d75f5c1574249b8726a00e0189dc4 P 0ddd486109f5434cace13d60d633ac04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:16.685942 8809 ts_tablet_manager.cc:1888] T 854d75f5c1574249b8726a00e0189dc4 P 0ddd486109f5434cace13d60d633ac04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:16.686036 8809 log.cc:1196] T 854d75f5c1574249b8726a00e0189dc4 P 0ddd486109f5434cace13d60d633ac04: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/wals/854d75f5c1574249b8726a00e0189dc4
I20240502 21:27:16.686421 8809 ts_tablet_manager.cc:1909] T 854d75f5c1574249b8726a00e0189dc4 P 0ddd486109f5434cace13d60d633ac04: Deleting consensus metadata
I20240502 21:27:16.687249 4285 catalog_manager.cc:4830] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): tablet 854d75f5c1574249b8726a00e0189dc4 (table rebalancer_dynamic_table_new_name [id=72456c0620c64c499b490c020bc04a27]) successfully deleted
I20240502 21:27:16.750159 8661 consensus_queue.cc:1035] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:16.756762 8768 consensus_queue.cc:1035] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:16.769636 4292 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:16.781803 7463 tablet_service.cc:1467] Processing CreateTablet for tablet 0a6b4539b5b04508ba8dcc0036d4c2b5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=103c5caf4f5a4426bc3fe4f853a748f8]), partition=
I20240502 21:27:16.781904 7595 tablet_service.cc:1467] Processing CreateTablet for tablet 0a6b4539b5b04508ba8dcc0036d4c2b5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=103c5caf4f5a4426bc3fe4f853a748f8]), partition=
I20240502 21:27:16.782229 7463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a6b4539b5b04508ba8dcc0036d4c2b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:16.782307 7595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a6b4539b5b04508ba8dcc0036d4c2b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:16.786448 4422 tablet_service.cc:1467] Processing CreateTablet for tablet 0a6b4539b5b04508ba8dcc0036d4c2b5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=103c5caf4f5a4426bc3fe4f853a748f8]), partition=
I20240502 21:27:16.786759 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a6b4539b5b04508ba8dcc0036d4c2b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:16.787129 8779 tablet_bootstrap.cc:492] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:16.788787 8779 tablet_bootstrap.cc:654] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:16.790679 8779 tablet_bootstrap.cc:492] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:16.790825 8779 ts_tablet_manager.cc:1367] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:27:16.791481 8779 raft_consensus.cc:348] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:16.791723 8779 raft_consensus.cc:374] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:16.791798 8779 raft_consensus.cc:729] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:16.792022 8779 consensus_queue.cc:260] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:16.793943 8779 ts_tablet_manager.cc:1398] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:16.797295 8814 tablet_bootstrap.cc:492] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:16.799299 8814 tablet_bootstrap.cc:654] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:16.807554 8816 tablet_bootstrap.cc:492] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:16.809017 8816 tablet_bootstrap.cc:654] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:16.813561 8816 tablet_bootstrap.cc:492] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:16.813730 8816 ts_tablet_manager.cc:1367] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 21:27:16.814368 8816 raft_consensus.cc:348] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:16.814641 8816 raft_consensus.cc:374] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:16.814752 8816 raft_consensus.cc:729] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:16.815016 8816 consensus_queue.cc:260] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:16.818552 8816 ts_tablet_manager.cc:1398] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:27:16.822217 8814 tablet_bootstrap.cc:492] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:16.822405 8814 ts_tablet_manager.cc:1367] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.018s
I20240502 21:27:16.823186 8814 raft_consensus.cc:348] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:16.823431 8814 raft_consensus.cc:374] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:16.823537 8814 raft_consensus.cc:729] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:16.823807 8814 consensus_queue.cc:260] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:16.824613 8814 ts_tablet_manager.cc:1398] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:16.949750 8767 raft_consensus.cc:482] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:16.949944 8767 raft_consensus.cc:504] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:16.950742 8767 leader_election.cc:288] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:16.951089 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a6b4539b5b04508ba8dcc0036d4c2b5" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:16.951346 7618 raft_consensus.cc:2454] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:16.951283 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a6b4539b5b04508ba8dcc0036d4c2b5" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:27:16.951545 4443 raft_consensus.cc:2454] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:16.951829 7418 leader_election.cc:302] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168; no voters:
I20240502 21:27:16.952111 8767 raft_consensus.cc:2781] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:16.952221 8767 raft_consensus.cc:482] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:16.952315 8767 raft_consensus.cc:3037] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:16.954768 8767 raft_consensus.cc:504] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:16.955396 8767 leader_election.cc:288] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Requested vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:16.955804 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a6b4539b5b04508ba8dcc0036d4c2b5" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:16.955895 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a6b4539b5b04508ba8dcc0036d4c2b5" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:27:16.956046 7618 raft_consensus.cc:3037] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:16.956136 4443 raft_consensus.cc:3037] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:16.958681 7618 raft_consensus.cc:2454] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:16.958760 4443 raft_consensus.cc:2454] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:16.959100 7418 leader_election.cc:302] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168; no voters:
I20240502 21:27:16.959424 8767 raft_consensus.cc:2781] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:16.960273 8767 raft_consensus.cc:686] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Becoming Leader. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Running, Role: LEADER
I20240502 21:27:16.960595 8767 consensus_queue.cc:237] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:16.963716 4300 catalog_manager.cc:5484] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a72286397c4edc9a63eca3412be168 (127.29.82.71). New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:17.039167 8651 consensus_queue.cc:1035] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:17.044068 8649 consensus_queue.cc:1035] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:17.092144 8651 consensus_queue.cc:1035] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:17.096325 8651 consensus_queue.cc:1035] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:17.099447 4300 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:17.105923 7616 raft_consensus.cc:1261] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:17.106333 4443 raft_consensus.cc:1261] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:17.106981 8819 consensus_queue.cc:1035] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:17.107852 8819 consensus_queue.cc:1035] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:17.111718 8829 tablet.cc:1722] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:17.112361 8829 tablet.cc:1620] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:17.113469 8831 tablet.cc:1722] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:17.114213 8831 tablet.cc:1620] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 0ddd486109f5434cace13d60d633ac04: MemRowSet was empty: no flush needed.
I20240502 21:27:17.120796 4292 catalog_manager.cc:5576] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223) does not have the latest schema for tablet 0a6b4539b5b04508ba8dcc0036d4c2b5 (table rebalancer_dynamic_table [id=103c5caf4f5a4426bc3fe4f853a748f8]). Expected version 1 got 0
I20240502 21:27:17.125152 8828 mvcc.cc:205] Tried to move back new op lower bound from 7023350731178831872 to 7023350730592432128. Current Snapshot: MvccSnapshot[applied={T|T < 7023350731178831872}]
I20240502 21:27:17.126654 4294 catalog_manager.cc:5576] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781) does not have the latest schema for tablet 0a6b4539b5b04508ba8dcc0036d4c2b5 (table rebalancer_dynamic_table [id=103c5caf4f5a4426bc3fe4f853a748f8]). Expected version 1 got 0
I20240502 21:27:17.127560 8832 tablet.cc:1722] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:17.130005 8832 tablet.cc:1620] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:17.131817 4300 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=103c5caf4f5a4426bc3fe4f853a748f8] alter complete (version 1)
I20240502 21:27:17.213328 4292 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:17.223984 4555 tablet_service.cc:1467] Processing CreateTablet for tablet 84c8bdb300064adc86a4b23e07d512d3 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=13f15c7aba6346b19da83861cfdb5f07]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:17.223977 4818 tablet_service.cc:1467] Processing CreateTablet for tablet 84c8bdb300064adc86a4b23e07d512d3 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=13f15c7aba6346b19da83861cfdb5f07]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:17.224182 7329 tablet_service.cc:1467] Processing CreateTablet for tablet 84c8bdb300064adc86a4b23e07d512d3 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=13f15c7aba6346b19da83861cfdb5f07]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:17.224467 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84c8bdb300064adc86a4b23e07d512d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:17.224481 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84c8bdb300064adc86a4b23e07d512d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:17.224591 7329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84c8bdb300064adc86a4b23e07d512d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:17.227965 4817 tablet_service.cc:1467] Processing CreateTablet for tablet 7b5db855b5b24c8b87d65c7973f90bbb (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=13f15c7aba6346b19da83861cfdb5f07]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:17.228376 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b5db855b5b24c8b87d65c7973f90bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:17.230934 8835 tablet_bootstrap.cc:492] T 84c8bdb300064adc86a4b23e07d512d3 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:17.232223 7328 tablet_service.cc:1467] Processing CreateTablet for tablet 7b5db855b5b24c8b87d65c7973f90bbb (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=13f15c7aba6346b19da83861cfdb5f07]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:17.232272 7195 tablet_service.cc:1467] Processing CreateTablet for tablet 7b5db855b5b24c8b87d65c7973f90bbb (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=13f15c7aba6346b19da83861cfdb5f07]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:17.232671 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b5db855b5b24c8b87d65c7973f90bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:17.232699 7195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b5db855b5b24c8b87d65c7973f90bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:17.235407 8835 tablet_bootstrap.cc:654] T 84c8bdb300064adc86a4b23e07d512d3 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:17.243849 4300 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:17.248257 7463 tablet_service.cc:1467] Processing CreateTablet for tablet 85fac458a4204f40a85d4ecba4718f58 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=13f15c7aba6346b19da83861cfdb5f07]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:17.248292 4818 tablet_service.cc:1467] Processing CreateTablet for tablet 85fac458a4204f40a85d4ecba4718f58 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=13f15c7aba6346b19da83861cfdb5f07]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:17.248749 7463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85fac458a4204f40a85d4ecba4718f58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:17.248800 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85fac458a4204f40a85d4ecba4718f58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:17.253044 8838 tablet_bootstrap.cc:492] T 7b5db855b5b24c8b87d65c7973f90bbb P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:17.256770 8838 tablet_bootstrap.cc:654] T 7b5db855b5b24c8b87d65c7973f90bbb P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:17.258639 8829 tablet.cc:1722] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:17.258960 8836 tablet_bootstrap.cc:492] T 84c8bdb300064adc86a4b23e07d512d3 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:17.259095 8829 tablet.cc:1620] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:17.260622 8836 tablet_bootstrap.cc:654] T 84c8bdb300064adc86a4b23e07d512d3 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:17.261723 7328 tablet_service.cc:1467] Processing CreateTablet for tablet 85fac458a4204f40a85d4ecba4718f58 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=13f15c7aba6346b19da83861cfdb5f07]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:17.262032 8831 tablet.cc:1722] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:17.262493 8831 tablet.cc:1620] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 0ddd486109f5434cace13d60d633ac04: MemRowSet was empty: no flush needed.
I20240502 21:27:17.263226 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85fac458a4204f40a85d4ecba4718f58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:17.267386 8816 tablet_bootstrap.cc:492] T 85fac458a4204f40a85d4ecba4718f58 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:17.269009 8816 tablet_bootstrap.cc:654] T 85fac458a4204f40a85d4ecba4718f58 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:17.271135 8832 tablet.cc:1722] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:17.271663 8832 tablet.cc:1620] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:17.276623 8840 tablet_bootstrap.cc:492] T 7b5db855b5b24c8b87d65c7973f90bbb P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:17.278996 4300 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=103c5caf4f5a4426bc3fe4f853a748f8] alter complete (version 2)
I20240502 21:27:17.280728 8835 tablet_bootstrap.cc:492] T 84c8bdb300064adc86a4b23e07d512d3 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:17.280895 8835 ts_tablet_manager.cc:1367] T 84c8bdb300064adc86a4b23e07d512d3 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.050s user 0.014s sys 0.004s
I20240502 21:27:17.281241 8840 tablet_bootstrap.cc:654] T 7b5db855b5b24c8b87d65c7973f90bbb P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:17.281788 8835 raft_consensus.cc:348] T 84c8bdb300064adc86a4b23e07d512d3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:17.282022 8835 raft_consensus.cc:374] T 84c8bdb300064adc86a4b23e07d512d3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:17.282127 8835 raft_consensus.cc:729] T 84c8bdb300064adc86a4b23e07d512d3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:17.282356 8835 consensus_queue.cc:260] T 84c8bdb300064adc86a4b23e07d512d3 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:17.282992 8835 ts_tablet_manager.cc:1398] T 84c8bdb300064adc86a4b23e07d512d3 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:17.283329 8835 tablet_bootstrap.cc:492] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:17.287227 8835 tablet_bootstrap.cc:654] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:17.287897 8836 tablet_bootstrap.cc:492] T 84c8bdb300064adc86a4b23e07d512d3 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:17.288103 8836 ts_tablet_manager.cc:1367] T 84c8bdb300064adc86a4b23e07d512d3 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.007s
I20240502 21:27:17.288306 8838 tablet_bootstrap.cc:492] T 7b5db855b5b24c8b87d65c7973f90bbb P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:17.288482 8838 ts_tablet_manager.cc:1367] T 7b5db855b5b24c8b87d65c7973f90bbb P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.006s
I20240502 21:27:17.289076 8836 raft_consensus.cc:348] T 84c8bdb300064adc86a4b23e07d512d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:17.289319 8836 raft_consensus.cc:374] T 84c8bdb300064adc86a4b23e07d512d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:17.289441 8836 raft_consensus.cc:729] T 84c8bdb300064adc86a4b23e07d512d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:17.289513 8838 raft_consensus.cc:348] T 7b5db855b5b24c8b87d65c7973f90bbb P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:17.289774 8838 raft_consensus.cc:374] T 7b5db855b5b24c8b87d65c7973f90bbb P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:17.289880 8838 raft_consensus.cc:729] T 7b5db855b5b24c8b87d65c7973f90bbb P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:17.290009 8836 consensus_queue.cc:260] T 84c8bdb300064adc86a4b23e07d512d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:17.290890 8836 ts_tablet_manager.cc:1398] T 84c8bdb300064adc86a4b23e07d512d3 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:17.292289 8838 consensus_queue.cc:260] T 7b5db855b5b24c8b87d65c7973f90bbb P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:17.293927 8838 ts_tablet_manager.cc:1398] T 7b5db855b5b24c8b87d65c7973f90bbb P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:27:17.294219 8838 tablet_bootstrap.cc:492] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:17.295743 8838 tablet_bootstrap.cc:654] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:17.307354 8838 tablet_bootstrap.cc:492] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:17.308491 8838 ts_tablet_manager.cc:1367] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.004s
I20240502 21:27:17.310942 8838 raft_consensus.cc:348] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:17.312294 8838 raft_consensus.cc:374] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:17.312572 8838 raft_consensus.cc:729] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:17.313055 8838 consensus_queue.cc:260] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:17.314173 8838 ts_tablet_manager.cc:1398] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 21:27:17.314919 8838 tablet_bootstrap.cc:492] T 85fac458a4204f40a85d4ecba4718f58 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:17.316751 8661 raft_consensus.cc:482] T 84c8bdb300064adc86a4b23e07d512d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:17.316936 8661 raft_consensus.cc:504] T 84c8bdb300064adc86a4b23e07d512d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:17.317157 8659 raft_consensus.cc:482] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:17.317317 8659 raft_consensus.cc:504] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:17.317942 8661 leader_election.cc:288] T 84c8bdb300064adc86a4b23e07d512d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:17.318303 8659 leader_election.cc:288] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:27:17.319509 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84c8bdb300064adc86a4b23e07d512d3" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:27:17.319646 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84c8bdb300064adc86a4b23e07d512d3" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:27:17.319754 4575 raft_consensus.cc:2454] T 84c8bdb300064adc86a4b23e07d512d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 0.
I20240502 21:27:17.319890 4838 raft_consensus.cc:2454] T 84c8bdb300064adc86a4b23e07d512d3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:27:17.320832 7285 leader_election.cc:302] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a736a8ebb164ce0a051c0fccd76c3b8, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:17.321250 4508 leader_election.cc:302] T 84c8bdb300064adc86a4b23e07d512d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:27:17.321734 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84c8bdb300064adc86a4b23e07d512d3" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:27:17.321966 4839 raft_consensus.cc:2454] T 84c8bdb300064adc86a4b23e07d512d3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 0.
I20240502 21:27:17.322240 8768 raft_consensus.cc:2781] T 84c8bdb300064adc86a4b23e07d512d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:17.322366 8768 raft_consensus.cc:482] T 84c8bdb300064adc86a4b23e07d512d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:17.322463 8768 raft_consensus.cc:3037] T 84c8bdb300064adc86a4b23e07d512d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:17.322743 8659 raft_consensus.cc:2781] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:17.322863 8659 raft_consensus.cc:482] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:17.322944 8659 raft_consensus.cc:3037] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:17.325381 8768 raft_consensus.cc:504] T 84c8bdb300064adc86a4b23e07d512d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:17.326025 8768 leader_election.cc:288] T 84c8bdb300064adc86a4b23e07d512d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:17.327523 8659 raft_consensus.cc:504] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:17.329200 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84c8bdb300064adc86a4b23e07d512d3" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:17.329457 7349 raft_consensus.cc:2379] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in current term 1: Already voted for candidate d0456c5e55a542128688904e003d403c in this term.
I20240502 21:27:17.330013 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84c8bdb300064adc86a4b23e07d512d3" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:17.330394 7350 raft_consensus.cc:2379] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in current term 1: Already voted for candidate d0456c5e55a542128688904e003d403c in this term.
I20240502 21:27:17.330870 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84c8bdb300064adc86a4b23e07d512d3" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:27:17.331146 4839 raft_consensus.cc:3037] T 84c8bdb300064adc86a4b23e07d512d3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:17.339308 4839 raft_consensus.cc:2454] T 84c8bdb300064adc86a4b23e07d512d3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
I20240502 21:27:17.341506 8659 leader_election.cc:288] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Requested vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:27:17.341719 8659 raft_consensus.cc:482] T 7b5db855b5b24c8b87d65c7973f90bbb P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:17.341862 8659 raft_consensus.cc:504] T 7b5db855b5b24c8b87d65c7973f90bbb P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:17.342309 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84c8bdb300064adc86a4b23e07d512d3" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:27:17.342563 4575 raft_consensus.cc:2379] T 84c8bdb300064adc86a4b23e07d512d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d0456c5e55a542128688904e003d403c in current term 1: Already voted for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in this term.
I20240502 21:27:17.342669 8659 leader_election.cc:288] T 7b5db855b5b24c8b87d65c7973f90bbb P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:17.343071 7281 leader_election.cc:302] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:17.344125 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b5db855b5b24c8b87d65c7973f90bbb" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
W20240502 21:27:17.345479 7281 leader_election.cc:341] T 7b5db855b5b24c8b87d65c7973f90bbb P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:17.345799 8849 raft_consensus.cc:2781] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:17.345990 8849 raft_consensus.cc:686] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Becoming Leader. State: Replica: d0456c5e55a542128688904e003d403c, State: Running, Role: LEADER
I20240502 21:27:17.346367 8849 consensus_queue.cc:237] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:17.360723 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84c8bdb300064adc86a4b23e07d512d3" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:27:17.361141 4839 raft_consensus.cc:2379] T 84c8bdb300064adc86a4b23e07d512d3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in current term 1: Already voted for candidate d0456c5e55a542128688904e003d403c in this term.
I20240502 21:27:17.362237 4508 leader_election.cc:302] T 84c8bdb300064adc86a4b23e07d512d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a736a8ebb164ce0a051c0fccd76c3b8; no voters: 73a1f2ca1f544de9b0e3b37015f85328, d0456c5e55a542128688904e003d403c
I20240502 21:27:17.362848 8768 raft_consensus.cc:2726] T 84c8bdb300064adc86a4b23e07d512d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 21:27:17.366775 8838 tablet_bootstrap.cc:654] T 85fac458a4204f40a85d4ecba4718f58 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:17.369554 8838 tablet_bootstrap.cc:492] T 85fac458a4204f40a85d4ecba4718f58 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:17.369957 8838 ts_tablet_manager.cc:1367] T 85fac458a4204f40a85d4ecba4718f58 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.000s
I20240502 21:27:17.370925 8838 raft_consensus.cc:348] T 85fac458a4204f40a85d4ecba4718f58 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:17.378192 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b5db855b5b24c8b87d65c7973f90bbb" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:17.388252 4300 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:17.394479 8838 raft_consensus.cc:374] T 85fac458a4204f40a85d4ecba4718f58 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:17.399693 8838 raft_consensus.cc:729] T 85fac458a4204f40a85d4ecba4718f58 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:17.401180 8838 consensus_queue.cc:260] T 85fac458a4204f40a85d4ecba4718f58 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:17.402081 8829 tablet.cc:1722] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:17.402518 8838 ts_tablet_manager.cc:1398] T 85fac458a4204f40a85d4ecba4718f58 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
I20240502 21:27:17.402926 8832 tablet.cc:1722] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240502 21:27:17.407686 7283 leader_election.cc:341] T 7b5db855b5b24c8b87d65c7973f90bbb P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:17.408406 8831 tablet.cc:1722] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:17.408488 7283 leader_election.cc:302] T 7b5db855b5b24c8b87d65c7973f90bbb P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0456c5e55a542128688904e003d403c; no voters: 39981a8dfe94462b9a73ba489af4c714, 73a1f2ca1f544de9b0e3b37015f85328
I20240502 21:27:17.409319 8849 raft_consensus.cc:2726] T 7b5db855b5b24c8b87d65c7973f90bbb P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:27:17.413829 4292 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=103c5caf4f5a4426bc3fe4f853a748f8] alter complete (version 3)
I20240502 21:27:17.414974 4291 catalog_manager.cc:5484] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c reported cstate change: term changed from 0 to 1, leader changed from <none> to d0456c5e55a542128688904e003d403c (127.29.82.70). New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:17.392733 8816 tablet_bootstrap.cc:492] T 85fac458a4204f40a85d4ecba4718f58 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:17.438532 8816 ts_tablet_manager.cc:1367] T 85fac458a4204f40a85d4ecba4718f58 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.171s user 0.012s sys 0.000s
I20240502 21:27:17.443208 8840 tablet_bootstrap.cc:492] T 7b5db855b5b24c8b87d65c7973f90bbb P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:17.443390 8840 ts_tablet_manager.cc:1367] T 7b5db855b5b24c8b87d65c7973f90bbb P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.167s user 0.009s sys 0.023s
I20240502 21:27:17.444289 8840 raft_consensus.cc:348] T 7b5db855b5b24c8b87d65c7973f90bbb P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:17.444514 8840 raft_consensus.cc:374] T 7b5db855b5b24c8b87d65c7973f90bbb P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:17.444612 8840 raft_consensus.cc:729] T 7b5db855b5b24c8b87d65c7973f90bbb P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:17.445120 8840 consensus_queue.cc:260] T 7b5db855b5b24c8b87d65c7973f90bbb P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:17.446363 8816 raft_consensus.cc:348] T 85fac458a4204f40a85d4ecba4718f58 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:17.446745 8816 raft_consensus.cc:374] T 85fac458a4204f40a85d4ecba4718f58 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:17.446897 8816 raft_consensus.cc:729] T 85fac458a4204f40a85d4ecba4718f58 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:17.447269 8840 ts_tablet_manager.cc:1398] T 7b5db855b5b24c8b87d65c7973f90bbb P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:17.447445 8816 consensus_queue.cc:260] T 85fac458a4204f40a85d4ecba4718f58 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:17.459667 8816 ts_tablet_manager.cc:1398] T 85fac458a4204f40a85d4ecba4718f58 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 21:27:17.461833 8835 tablet_bootstrap.cc:492] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:17.467355 8835 ts_tablet_manager.cc:1367] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.184s user 0.041s sys 0.084s
I20240502 21:27:17.468351 8835 raft_consensus.cc:348] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:17.468654 8835 raft_consensus.cc:374] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:17.468748 8835 raft_consensus.cc:729] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:17.469004 8835 consensus_queue.cc:260] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:17.471101 8024 raft_consensus.cc:482] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:17.471238 8024 raft_consensus.cc:504] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:17.472112 8024 leader_election.cc:288] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:17.473101 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b5db855b5b24c8b87d65c7973f90bbb" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:17.473316 7215 raft_consensus.cc:2454] T 7b5db855b5b24c8b87d65c7973f90bbb P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:27:17.473357 8835 ts_tablet_manager.cc:1398] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 21:27:17.473518 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b5db855b5b24c8b87d65c7973f90bbb" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:17.473755 7350 raft_consensus.cc:2454] T 7b5db855b5b24c8b87d65c7973f90bbb P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:27:17.474045 4772 leader_election.cc:302] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:17.474941 8561 raft_consensus.cc:2781] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:17.475023 8561 raft_consensus.cc:482] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:17.475108 8561 raft_consensus.cc:3037] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:17.477450 8835 tablet_bootstrap.cc:492] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:17.478003 8561 raft_consensus.cc:504] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:17.478646 8561 leader_election.cc:288] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:17.479166 8835 tablet_bootstrap.cc:654] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:17.479700 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b5db855b5b24c8b87d65c7973f90bbb" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:17.479985 7350 raft_consensus.cc:3037] T 7b5db855b5b24c8b87d65c7973f90bbb P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:17.480037 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b5db855b5b24c8b87d65c7973f90bbb" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:17.480237 7215 raft_consensus.cc:3037] T 7b5db855b5b24c8b87d65c7973f90bbb P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:17.483539 7215 raft_consensus.cc:2454] T 7b5db855b5b24c8b87d65c7973f90bbb P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:27:17.483981 4772 leader_election.cc:302] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:17.484534 8024 raft_consensus.cc:2781] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:17.484680 8024 raft_consensus.cc:686] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:27:17.484687 7350 raft_consensus.cc:2454] T 7b5db855b5b24c8b87d65c7973f90bbb P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:27:17.484928 8024 consensus_queue.cc:237] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:17.490860 8835 tablet_bootstrap.cc:492] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:17.491143 8835 ts_tablet_manager.cc:1367] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.005s
I20240502 21:27:17.492553 8835 raft_consensus.cc:348] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:17.492936 8835 raft_consensus.cc:374] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:17.493119 8835 raft_consensus.cc:729] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:17.493512 8835 consensus_queue.cc:260] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:17.494377 8835 ts_tablet_manager.cc:1398] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:17.498463 4291 catalog_manager.cc:5484] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:17.535290 4300 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:17.535559 4300 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:17.538801 4300 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 0a6b4539b5b04508ba8dcc0036d4c2b5
I20240502 21:27:17.539669 7595 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a6b4539b5b04508ba8dcc0036d4c2b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:17 UTC) from {username='slave'} at 127.0.0.1:44444
I20240502 21:27:17.539762 7463 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a6b4539b5b04508ba8dcc0036d4c2b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:17 UTC) from {username='slave'} at 127.0.0.1:37182
I20240502 21:27:17.539901 4422 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a6b4539b5b04508ba8dcc0036d4c2b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:17 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:17.540390 8856 tablet_replica.cc:330] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 0ddd486109f5434cace13d60d633ac04: stopping tablet replica
I20240502 21:27:17.540719 8857 tablet_replica.cc:330] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:27:17.540779 8856 raft_consensus.cc:2229] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:17.540951 8857 raft_consensus.cc:2229] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:17.541091 8856 raft_consensus.cc:2258] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:17.541368 8857 raft_consensus.cc:2258] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:17.542299 8857 ts_tablet_manager.cc:1875] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:17.543937 8856 ts_tablet_manager.cc:1875] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 0ddd486109f5434cace13d60d633ac04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:17.544034 8858 tablet_replica.cc:330] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:17.544320 8858 raft_consensus.cc:2229] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:17.544589 8858 raft_consensus.cc:2258] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:17.545454 8858 ts_tablet_manager.cc:1875] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:17.548482 8857 ts_tablet_manager.cc:1888] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:17.548621 8857 log.cc:1196] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/0a6b4539b5b04508ba8dcc0036d4c2b5
I20240502 21:27:17.549628 8856 ts_tablet_manager.cc:1888] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 0ddd486109f5434cace13d60d633ac04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:17.549726 8857 ts_tablet_manager.cc:1909] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 31a72286397c4edc9a63eca3412be168: Deleting consensus metadata
I20240502 21:27:17.549876 8856 log.cc:1196] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 0ddd486109f5434cace13d60d633ac04: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/wals/0a6b4539b5b04508ba8dcc0036d4c2b5
I20240502 21:27:17.550432 8856 ts_tablet_manager.cc:1909] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 0ddd486109f5434cace13d60d633ac04: Deleting consensus metadata
I20240502 21:27:17.551764 4285 catalog_manager.cc:4830] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): tablet 0a6b4539b5b04508ba8dcc0036d4c2b5 (table rebalancer_dynamic_table_new_name [id=103c5caf4f5a4426bc3fe4f853a748f8]) successfully deleted
I20240502 21:27:17.552086 8858 ts_tablet_manager.cc:1888] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:17.552253 8858 log.cc:1196] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/0a6b4539b5b04508ba8dcc0036d4c2b5
I20240502 21:27:17.552754 8858 ts_tablet_manager.cc:1909] T 0a6b4539b5b04508ba8dcc0036d4c2b5 P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:17.553890 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet 0a6b4539b5b04508ba8dcc0036d4c2b5 (table rebalancer_dynamic_table_new_name [id=103c5caf4f5a4426bc3fe4f853a748f8]) successfully deleted
I20240502 21:27:17.554162 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet 0a6b4539b5b04508ba8dcc0036d4c2b5 (table rebalancer_dynamic_table_new_name [id=103c5caf4f5a4426bc3fe4f853a748f8]) successfully deleted
I20240502 21:27:17.585986 8561 raft_consensus.cc:482] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:17.586227 8561 raft_consensus.cc:504] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:17.587133 8561 leader_election.cc:288] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:17.587641 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85fac458a4204f40a85d4ecba4718f58" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:17.587878 7484 raft_consensus.cc:2454] T 85fac458a4204f40a85d4ecba4718f58 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:27:17.588037 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85fac458a4204f40a85d4ecba4718f58" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:17.588297 7350 raft_consensus.cc:2454] T 85fac458a4204f40a85d4ecba4718f58 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:27:17.588742 4772 leader_election.cc:302] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:17.589208 8561 raft_consensus.cc:2781] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:17.589319 8561 raft_consensus.cc:482] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:17.589433 8561 raft_consensus.cc:3037] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:17.591368 8561 raft_consensus.cc:504] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:17.592068 8561 leader_election.cc:288] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:17.592365 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85fac458a4204f40a85d4ecba4718f58" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:17.592554 7484 raft_consensus.cc:3037] T 85fac458a4204f40a85d4ecba4718f58 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:17.592552 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85fac458a4204f40a85d4ecba4718f58" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:17.592758 7350 raft_consensus.cc:3037] T 85fac458a4204f40a85d4ecba4718f58 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:17.594478 7484 raft_consensus.cc:2454] T 85fac458a4204f40a85d4ecba4718f58 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:27:17.594880 4770 leader_election.cc:302] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:17.595196 8561 raft_consensus.cc:2781] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:17.595386 8561 raft_consensus.cc:686] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:27:17.595497 7350 raft_consensus.cc:2454] T 85fac458a4204f40a85d4ecba4718f58 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:27:17.595758 8561 consensus_queue.cc:237] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:17.599534 4300 catalog_manager.cc:5484] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:17.641515 4300 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:17.649112 7595 tablet_service.cc:1467] Processing CreateTablet for tablet 5255cf9647474d8bbb4141cbac67ba44 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e0549c7b124b4f83866eacba79670952]), partition=
I20240502 21:27:17.649551 7595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5255cf9647474d8bbb4141cbac67ba44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:17.649627 7195 tablet_service.cc:1467] Processing CreateTablet for tablet 5255cf9647474d8bbb4141cbac67ba44 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e0549c7b124b4f83866eacba79670952]), partition=
I20240502 21:27:17.649994 7195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5255cf9647474d8bbb4141cbac67ba44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:17.652365 7463 tablet_service.cc:1467] Processing CreateTablet for tablet 5255cf9647474d8bbb4141cbac67ba44 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e0549c7b124b4f83866eacba79670952]), partition=
I20240502 21:27:17.652740 7463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5255cf9647474d8bbb4141cbac67ba44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:17.656277 8840 tablet_bootstrap.cc:492] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:17.657868 8840 tablet_bootstrap.cc:654] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:17.658059 8816 tablet_bootstrap.cc:492] T 5255cf9647474d8bbb4141cbac67ba44 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:17.659646 8816 tablet_bootstrap.cc:654] T 5255cf9647474d8bbb4141cbac67ba44 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:17.662573 8840 tablet_bootstrap.cc:492] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:17.662585 8816 tablet_bootstrap.cc:492] T 5255cf9647474d8bbb4141cbac67ba44 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:17.662751 8816 ts_tablet_manager.cc:1367] T 5255cf9647474d8bbb4141cbac67ba44 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:27:17.662750 8840 ts_tablet_manager.cc:1367] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20240502 21:27:17.663368 8840 raft_consensus.cc:348] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:17.663554 8840 raft_consensus.cc:374] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:17.663656 8840 raft_consensus.cc:729] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:17.663552 8816 raft_consensus.cc:348] T 5255cf9647474d8bbb4141cbac67ba44 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:17.663921 8816 raft_consensus.cc:374] T 5255cf9647474d8bbb4141cbac67ba44 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:17.663928 8840 consensus_queue.cc:260] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:17.664022 8816 raft_consensus.cc:729] T 5255cf9647474d8bbb4141cbac67ba44 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:17.664413 8840 ts_tablet_manager.cc:1398] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:27:17.664877 8860 tablet_bootstrap.cc:492] T 5255cf9647474d8bbb4141cbac67ba44 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:17.666579 8860 tablet_bootstrap.cc:654] T 5255cf9647474d8bbb4141cbac67ba44 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:17.666494 8816 consensus_queue.cc:260] T 5255cf9647474d8bbb4141cbac67ba44 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:17.667340 8816 ts_tablet_manager.cc:1398] T 5255cf9647474d8bbb4141cbac67ba44 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 21:27:17.675150 8860 tablet_bootstrap.cc:492] T 5255cf9647474d8bbb4141cbac67ba44 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:17.675309 8860 ts_tablet_manager.cc:1367] T 5255cf9647474d8bbb4141cbac67ba44 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240502 21:27:17.676056 8860 raft_consensus.cc:348] T 5255cf9647474d8bbb4141cbac67ba44 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:17.676298 8860 raft_consensus.cc:374] T 5255cf9647474d8bbb4141cbac67ba44 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:17.676414 8860 raft_consensus.cc:729] T 5255cf9647474d8bbb4141cbac67ba44 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:17.676627 8860 consensus_queue.cc:260] T 5255cf9647474d8bbb4141cbac67ba44 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:17.677207 8860 ts_tablet_manager.cc:1398] T 5255cf9647474d8bbb4141cbac67ba44 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:17.767347 8846 consensus_queue.cc:1035] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:17.785543 8849 consensus_queue.cc:1035] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:17.824599 8649 raft_consensus.cc:482] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:17.824771 8649 raft_consensus.cc:504] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:17.825443 8649 leader_election.cc:288] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:17.826041 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5255cf9647474d8bbb4141cbac67ba44" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:17.826028 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5255cf9647474d8bbb4141cbac67ba44" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:17.826278 7618 raft_consensus.cc:2454] T 5255cf9647474d8bbb4141cbac67ba44 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:17.826297 7484 raft_consensus.cc:2454] T 5255cf9647474d8bbb4141cbac67ba44 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:17.826650 7148 leader_election.cc:302] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:17.827057 8651 raft_consensus.cc:2781] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:17.827172 8651 raft_consensus.cc:482] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:17.827293 8651 raft_consensus.cc:3037] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:17.829753 8651 raft_consensus.cc:504] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:17.830389 8651 leader_election.cc:288] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:17.830768 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5255cf9647474d8bbb4141cbac67ba44" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:17.830996 7618 raft_consensus.cc:3037] T 5255cf9647474d8bbb4141cbac67ba44 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:17.831272 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5255cf9647474d8bbb4141cbac67ba44" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:17.831480 7484 raft_consensus.cc:3037] T 5255cf9647474d8bbb4141cbac67ba44 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:17.833845 7618 raft_consensus.cc:2454] T 5255cf9647474d8bbb4141cbac67ba44 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:17.834362 7149 leader_election.cc:302] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:17.834589 7484 raft_consensus.cc:2454] T 5255cf9647474d8bbb4141cbac67ba44 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:17.834661 8651 raft_consensus.cc:2781] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:17.834921 8651 raft_consensus.cc:686] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:27:17.835171 8651 consensus_queue.cc:237] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:17.839427 4300 catalog_manager.cc:5484] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69). New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:17.950654 4300 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:17.956691 7618 raft_consensus.cc:1261] T 5255cf9647474d8bbb4141cbac67ba44 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:17.956699 7484 raft_consensus.cc:1261] T 5255cf9647474d8bbb4141cbac67ba44 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:17.957363 8649 consensus_queue.cc:1035] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:17.957681 8651 consensus_queue.cc:1035] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:17.963920 4300 catalog_manager.cc:5576] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223) does not have the latest schema for tablet 5255cf9647474d8bbb4141cbac67ba44 (table rebalancer_dynamic_table [id=e0549c7b124b4f83866eacba79670952]). Expected version 1 got 0
I20240502 21:27:17.964699 4291 catalog_manager.cc:5576] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913) does not have the latest schema for tablet 5255cf9647474d8bbb4141cbac67ba44 (table rebalancer_dynamic_table [id=e0549c7b124b4f83866eacba79670952]). Expected version 1 got 0
I20240502 21:27:17.965984 8876 tablet.cc:1722] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:17.966481 8877 tablet.cc:1722] T 5255cf9647474d8bbb4141cbac67ba44 P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:17.966683 8876 tablet.cc:1620] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:17.967027 8877 tablet.cc:1620] T 5255cf9647474d8bbb4141cbac67ba44 P 0ddd486109f5434cace13d60d633ac04: MemRowSet was empty: no flush needed.
I20240502 21:27:17.969763 8872 mvcc.cc:205] Tried to move back new op lower bound from 7023350734664433664 to 7023350734175748096. Current Snapshot: MvccSnapshot[applied={T|T < 7023350734664433664}]
I20240502 21:27:17.971773 8878 tablet.cc:1722] T 5255cf9647474d8bbb4141cbac67ba44 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:17.972569 8878 tablet.cc:1620] T 5255cf9647474d8bbb4141cbac67ba44 P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:17.975992 4291 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=e0549c7b124b4f83866eacba79670952] alter complete (version 1)
I20240502 21:27:17.978287 4291 catalog_manager.cc:5576] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913) does not have the latest schema for tablet 5255cf9647474d8bbb4141cbac67ba44 (table rebalancer_dynamic_table [id=e0549c7b124b4f83866eacba79670952]). Expected version 1 got 0
I20240502 21:27:18.022166 8024 consensus_queue.cc:1035] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:18.029186 8024 consensus_queue.cc:1035] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:18.080974 4292 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:18.088599 8876 tablet.cc:1722] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:18.089187 8876 tablet.cc:1620] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714: MemRowSet was empty: no flush needed.
I20240502 21:27:18.089543 8878 tablet.cc:1722] T 5255cf9647474d8bbb4141cbac67ba44 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:18.090121 8878 tablet.cc:1620] T 5255cf9647474d8bbb4141cbac67ba44 P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:18.090867 8877 tablet.cc:1722] T 5255cf9647474d8bbb4141cbac67ba44 P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:18.091308 8877 tablet.cc:1620] T 5255cf9647474d8bbb4141cbac67ba44 P 0ddd486109f5434cace13d60d633ac04: MemRowSet was empty: no flush needed.
I20240502 21:27:18.100692 4300 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=e0549c7b124b4f83866eacba79670952] alter complete (version 2)
I20240502 21:27:18.110559 4300 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:18.120498 4818 tablet_service.cc:1467] Processing CreateTablet for tablet f4cf59f48e874f42aed76d694246e7d3 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=9c0b534f9084406c81e37e510fcd71d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:18.121026 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4cf59f48e874f42aed76d694246e7d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:18.121165 4422 tablet_service.cc:1467] Processing CreateTablet for tablet 1bddb5e40d464de082c1d586618c86d7 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=9c0b534f9084406c81e37e510fcd71d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:18.121630 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bddb5e40d464de082c1d586618c86d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:18.121779 4685 tablet_service.cc:1467] Processing CreateTablet for tablet f4cf59f48e874f42aed76d694246e7d3 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=9c0b534f9084406c81e37e510fcd71d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:18.122213 4685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4cf59f48e874f42aed76d694246e7d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:18.124246 7595 tablet_service.cc:1467] Processing CreateTablet for tablet a4c36adfc86141fbb8724d5a6a798c26 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=9c0b534f9084406c81e37e510fcd71d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:18.124689 7595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c36adfc86141fbb8724d5a6a798c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:18.125481 4686 tablet_service.cc:1467] Processing CreateTablet for tablet a4c36adfc86141fbb8724d5a6a798c26 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=9c0b534f9084406c81e37e510fcd71d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:18.125914 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c36adfc86141fbb8724d5a6a798c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:18.126633 4553 tablet_service.cc:1467] Processing CreateTablet for tablet f4cf59f48e874f42aed76d694246e7d3 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=9c0b534f9084406c81e37e510fcd71d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:18.126734 4555 tablet_service.cc:1467] Processing CreateTablet for tablet a4c36adfc86141fbb8724d5a6a798c26 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=9c0b534f9084406c81e37e510fcd71d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:18.127140 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4cf59f48e874f42aed76d694246e7d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:18.127379 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c36adfc86141fbb8724d5a6a798c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:18.128674 4684 tablet_service.cc:1467] Processing CreateTablet for tablet 1bddb5e40d464de082c1d586618c86d7 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=9c0b534f9084406c81e37e510fcd71d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:18.129042 4684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bddb5e40d464de082c1d586618c86d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:18.131103 7597 tablet_service.cc:1467] Processing CreateTablet for tablet 1bddb5e40d464de082c1d586618c86d7 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=9c0b534f9084406c81e37e510fcd71d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:18.131536 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bddb5e40d464de082c1d586618c86d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:18.138636 8860 tablet_bootstrap.cc:492] T a4c36adfc86141fbb8724d5a6a798c26 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:18.140432 8860 tablet_bootstrap.cc:654] T a4c36adfc86141fbb8724d5a6a798c26 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:18.148171 8885 tablet_bootstrap.cc:492] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:18.150673 8885 tablet_bootstrap.cc:654] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:18.151139 8881 tablet_bootstrap.cc:492] T a4c36adfc86141fbb8724d5a6a798c26 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:18.156002 8024 consensus_queue.cc:1035] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:18.156752 8860 tablet_bootstrap.cc:492] T a4c36adfc86141fbb8724d5a6a798c26 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:18.156842 8883 tablet_bootstrap.cc:492] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:18.157050 8860 ts_tablet_manager.cc:1367] T a4c36adfc86141fbb8724d5a6a798c26 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.004s
I20240502 21:27:18.159345 8883 tablet_bootstrap.cc:654] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:18.162842 8885 tablet_bootstrap.cc:492] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:27:18.163038 8885 ts_tablet_manager.cc:1367] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20240502 21:27:18.164296 8881 tablet_bootstrap.cc:654] T a4c36adfc86141fbb8724d5a6a798c26 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:18.165889 8561 consensus_queue.cc:1035] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:18.166234 8885 raft_consensus.cc:348] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:18.166597 8885 raft_consensus.cc:374] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:18.166770 8885 raft_consensus.cc:729] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:18.167131 8885 consensus_queue.cc:260] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:18.172499 8885 ts_tablet_manager.cc:1398] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 21:27:18.176901 8885 tablet_bootstrap.cc:492] T 1bddb5e40d464de082c1d586618c86d7 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:18.181167 8885 tablet_bootstrap.cc:654] T 1bddb5e40d464de082c1d586618c86d7 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:18.183626 8860 raft_consensus.cc:348] T a4c36adfc86141fbb8724d5a6a798c26 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:18.184069 8860 raft_consensus.cc:374] T a4c36adfc86141fbb8724d5a6a798c26 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:18.184211 8860 raft_consensus.cc:729] T a4c36adfc86141fbb8724d5a6a798c26 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:18.184535 8860 consensus_queue.cc:260] T a4c36adfc86141fbb8724d5a6a798c26 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:18.186518 8860 ts_tablet_manager.cc:1398] T a4c36adfc86141fbb8724d5a6a798c26 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:18.186921 8860 tablet_bootstrap.cc:492] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:18.190519 8884 tablet_bootstrap.cc:492] T f4cf59f48e874f42aed76d694246e7d3 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:18.194258 8860 tablet_bootstrap.cc:654] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:18.195036 8884 tablet_bootstrap.cc:654] T f4cf59f48e874f42aed76d694246e7d3 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:18.196110 8885 tablet_bootstrap.cc:492] T 1bddb5e40d464de082c1d586618c86d7 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:27:18.196373 8885 ts_tablet_manager.cc:1367] T 1bddb5e40d464de082c1d586618c86d7 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.000s
I20240502 21:27:18.197396 8885 raft_consensus.cc:348] T 1bddb5e40d464de082c1d586618c86d7 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:18.197763 8885 raft_consensus.cc:374] T 1bddb5e40d464de082c1d586618c86d7 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:18.197911 8885 raft_consensus.cc:729] T 1bddb5e40d464de082c1d586618c86d7 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:18.198315 8885 consensus_queue.cc:260] T 1bddb5e40d464de082c1d586618c86d7 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:18.202744 8885 ts_tablet_manager.cc:1398] T 1bddb5e40d464de082c1d586618c86d7 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 21:27:18.203419 8883 tablet_bootstrap.cc:492] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:18.203681 8883 ts_tablet_manager.cc:1367] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.047s user 0.010s sys 0.022s
I20240502 21:27:18.204766 8883 raft_consensus.cc:348] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:18.205030 8883 raft_consensus.cc:374] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:18.205129 8883 raft_consensus.cc:729] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:18.204998 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:18.205423 8883 consensus_queue.cc:260] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:18.206037 8883 ts_tablet_manager.cc:1398] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:18.206506 8884 tablet_bootstrap.cc:492] T f4cf59f48e874f42aed76d694246e7d3 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:18.206651 8884 ts_tablet_manager.cc:1367] T f4cf59f48e874f42aed76d694246e7d3 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20240502 21:27:18.210729 8885 tablet_bootstrap.cc:492] T f4cf59f48e874f42aed76d694246e7d3 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:18.212795 8885 tablet_bootstrap.cc:654] T f4cf59f48e874f42aed76d694246e7d3 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:18.214584 8860 tablet_bootstrap.cc:492] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:18.214741 8860 ts_tablet_manager.cc:1367] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.003s
I20240502 21:27:18.215610 8860 raft_consensus.cc:348] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:18.215833 8860 raft_consensus.cc:374] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:18.216461 8860 raft_consensus.cc:729] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:18.216737 8860 consensus_queue.cc:260] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:18.217005 8876 tablet.cc:1722] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:18.217873 8892 raft_consensus.cc:482] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:18.218036 8892 raft_consensus.cc:504] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:18.219075 8877 tablet.cc:1722] T 5255cf9647474d8bbb4141cbac67ba44 P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:18.217216 8884 raft_consensus.cc:348] T f4cf59f48e874f42aed76d694246e7d3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:18.220804 8884 raft_consensus.cc:374] T f4cf59f48e874f42aed76d694246e7d3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:18.220965 8884 raft_consensus.cc:729] T f4cf59f48e874f42aed76d694246e7d3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:18.221093 8860 ts_tablet_manager.cc:1398] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 21:27:18.221768 8884 consensus_queue.cc:260] T f4cf59f48e874f42aed76d694246e7d3 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:18.222816 8884 ts_tablet_manager.cc:1398] T f4cf59f48e874f42aed76d694246e7d3 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20240502 21:27:18.226753 8878 tablet.cc:1722] T 5255cf9647474d8bbb4141cbac67ba44 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:18.235138 4292 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=e0549c7b124b4f83866eacba79670952] alter complete (version 3)
W20240502 21:27:18.241852 4378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.66:35781, user_credentials={real_user=slave}} blocked reactor thread for 65465.1us
I20240502 21:27:18.244704 8892 leader_election.cc:288] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:27:18.245165 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bddb5e40d464de082c1d586618c86d7" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:27:18.245417 4706 raft_consensus.cc:2454] T 1bddb5e40d464de082c1d586618c86d7 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:18.245951 8818 raft_consensus.cc:482] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:18.246120 8818 raft_consensus.cc:504] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:18.247073 8818 leader_election.cc:288] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:18.247464 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c36adfc86141fbb8724d5a6a798c26" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
W20240502 21:27:18.248066 4642 leader_election.cc:341] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:18.249364 7616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c36adfc86141fbb8724d5a6a798c26" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:18.249646 7616 raft_consensus.cc:2454] T a4c36adfc86141fbb8724d5a6a798c26 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:27:18.250053 4640 leader_election.cc:302] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 398c3c8f3195492c900439ad5e1931bb; no voters: 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:27:18.250667 8818 raft_consensus.cc:2781] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:18.250794 8818 raft_consensus.cc:482] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:18.250901 8818 raft_consensus.cc:3037] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:18.252316 8881 tablet_bootstrap.cc:492] T a4c36adfc86141fbb8724d5a6a798c26 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:18.252492 8881 ts_tablet_manager.cc:1367] T a4c36adfc86141fbb8724d5a6a798c26 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.101s user 0.000s sys 0.003s
I20240502 21:27:18.253388 8881 raft_consensus.cc:348] T a4c36adfc86141fbb8724d5a6a798c26 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:18.253612 8881 raft_consensus.cc:374] T a4c36adfc86141fbb8724d5a6a798c26 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:18.253721 8881 raft_consensus.cc:729] T a4c36adfc86141fbb8724d5a6a798c26 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:18.253988 8881 consensus_queue.cc:260] T a4c36adfc86141fbb8724d5a6a798c26 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:18.254667 8818 raft_consensus.cc:504] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:18.255556 8818 leader_election.cc:288] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Requested vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:18.255697 8881 ts_tablet_manager.cc:1398] T a4c36adfc86141fbb8724d5a6a798c26 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:18.255862 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c36adfc86141fbb8724d5a6a798c26" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:27:18.256033 8881 tablet_bootstrap.cc:492] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:18.256072 4574 raft_consensus.cc:3037] T a4c36adfc86141fbb8724d5a6a798c26 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:18.256136 7616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c36adfc86141fbb8724d5a6a798c26" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:18.256356 7616 raft_consensus.cc:3037] T a4c36adfc86141fbb8724d5a6a798c26 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:18.258035 8881 tablet_bootstrap.cc:654] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:18.258566 4574 raft_consensus.cc:2454] T a4c36adfc86141fbb8724d5a6a798c26 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:27:18.259218 4642 leader_election.cc:302] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:27:18.259735 8818 raft_consensus.cc:2781] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:18.259958 7616 raft_consensus.cc:2454] T a4c36adfc86141fbb8724d5a6a798c26 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:27:18.260800 8885 tablet_bootstrap.cc:492] T f4cf59f48e874f42aed76d694246e7d3 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:27:18.261019 8885 ts_tablet_manager.cc:1367] T f4cf59f48e874f42aed76d694246e7d3 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.050s user 0.005s sys 0.000s
I20240502 21:27:18.262524 8885 raft_consensus.cc:348] T f4cf59f48e874f42aed76d694246e7d3 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:18.262799 8885 raft_consensus.cc:374] T f4cf59f48e874f42aed76d694246e7d3 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:18.262950 8885 raft_consensus.cc:729] T f4cf59f48e874f42aed76d694246e7d3 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:18.263038 7616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bddb5e40d464de082c1d586618c86d7" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:18.263242 7616 raft_consensus.cc:2454] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:18.263368 8885 consensus_queue.cc:260] T f4cf59f48e874f42aed76d694246e7d3 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:18.265554 4378 leader_election.cc:302] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:18.265959 8892 raft_consensus.cc:2781] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:18.266130 8892 raft_consensus.cc:482] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:18.266259 8892 raft_consensus.cc:3037] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:18.266338 8818 raft_consensus.cc:686] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Becoming Leader. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Running, Role: LEADER
I20240502 21:27:18.266705 8818 consensus_queue.cc:237] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:18.271811 8881 tablet_bootstrap.cc:492] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:18.271986 8881 ts_tablet_manager.cc:1367] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240502 21:27:18.272670 8881 raft_consensus.cc:348] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:18.272851 8881 raft_consensus.cc:374] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:18.272935 8881 raft_consensus.cc:729] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:18.272784 8892 raft_consensus.cc:504] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:18.273181 8881 consensus_queue.cc:260] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:18.274087 7616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bddb5e40d464de082c1d586618c86d7" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:18.274278 7616 raft_consensus.cc:3037] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:18.274278 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bddb5e40d464de082c1d586618c86d7" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:27:18.274514 8885 ts_tablet_manager.cc:1398] T f4cf59f48e874f42aed76d694246e7d3 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240502 21:27:18.274550 4706 raft_consensus.cc:3037] T 1bddb5e40d464de082c1d586618c86d7 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:18.277029 7616 raft_consensus.cc:2454] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:18.277596 4376 leader_election.cc:302] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:18.277935 8765 raft_consensus.cc:2781] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:18.278074 8765 raft_consensus.cc:686] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:27:18.278357 8765 consensus_queue.cc:237] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:18.279816 8881 ts_tablet_manager.cc:1398] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 21:27:18.280661 4706 raft_consensus.cc:2454] T 1bddb5e40d464de082c1d586618c86d7 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:18.281164 8892 leader_election.cc:288] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:27:18.285554 4300 catalog_manager.cc:5484] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:18.289597 4291 catalog_manager.cc:5484] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 398c3c8f3195492c900439ad5e1931bb (127.29.82.67). New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:18.341377 4291 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:18.341800 4291 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:18.345572 4291 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 5255cf9647474d8bbb4141cbac67ba44
I20240502 21:27:18.346602 7595 tablet_service.cc:1514] Processing DeleteTablet for tablet 5255cf9647474d8bbb4141cbac67ba44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:18 UTC) from {username='slave'} at 127.0.0.1:44444
I20240502 21:27:18.347280 7463 tablet_service.cc:1514] Processing DeleteTablet for tablet 5255cf9647474d8bbb4141cbac67ba44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:18 UTC) from {username='slave'} at 127.0.0.1:37182
I20240502 21:27:18.347713 8905 tablet_replica.cc:330] T 5255cf9647474d8bbb4141cbac67ba44 P 0ddd486109f5434cace13d60d633ac04: stopping tablet replica
I20240502 21:27:18.347951 8906 tablet_replica.cc:330] T 5255cf9647474d8bbb4141cbac67ba44 P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:27:18.348016 7195 tablet_service.cc:1514] Processing DeleteTablet for tablet 5255cf9647474d8bbb4141cbac67ba44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:18 UTC) from {username='slave'} at 127.0.0.1:44652
I20240502 21:27:18.348153 8905 raft_consensus.cc:2229] T 5255cf9647474d8bbb4141cbac67ba44 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:18.348398 8906 raft_consensus.cc:2229] T 5255cf9647474d8bbb4141cbac67ba44 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:18.348448 8905 raft_consensus.cc:2258] T 5255cf9647474d8bbb4141cbac67ba44 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:18.349042 8906 raft_consensus.cc:2258] T 5255cf9647474d8bbb4141cbac67ba44 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:18.349965 8906 ts_tablet_manager.cc:1875] T 5255cf9647474d8bbb4141cbac67ba44 P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:18.355841 8905 ts_tablet_manager.cc:1875] T 5255cf9647474d8bbb4141cbac67ba44 P 0ddd486109f5434cace13d60d633ac04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:18.360080 8907 tablet_replica.cc:330] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:27:18.360579 8907 raft_consensus.cc:2229] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:18.360963 8907 raft_consensus.cc:2258] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:18.361739 8907 ts_tablet_manager.cc:1875] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:18.362723 8905 ts_tablet_manager.cc:1888] T 5255cf9647474d8bbb4141cbac67ba44 P 0ddd486109f5434cace13d60d633ac04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:18.362874 8905 log.cc:1196] T 5255cf9647474d8bbb4141cbac67ba44 P 0ddd486109f5434cace13d60d633ac04: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/wals/5255cf9647474d8bbb4141cbac67ba44
I20240502 21:27:18.363365 8905 ts_tablet_manager.cc:1909] T 5255cf9647474d8bbb4141cbac67ba44 P 0ddd486109f5434cace13d60d633ac04: Deleting consensus metadata
I20240502 21:27:18.364688 4285 catalog_manager.cc:4830] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): tablet 5255cf9647474d8bbb4141cbac67ba44 (table rebalancer_dynamic_table_new_name [id=e0549c7b124b4f83866eacba79670952]) successfully deleted
I20240502 21:27:18.367466 8907 ts_tablet_manager.cc:1888] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:18.367605 8907 log.cc:1196] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/5255cf9647474d8bbb4141cbac67ba44
I20240502 21:27:18.368077 8907 ts_tablet_manager.cc:1909] T 5255cf9647474d8bbb4141cbac67ba44 P 39981a8dfe94462b9a73ba489af4c714: Deleting consensus metadata
I20240502 21:27:18.369203 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet 5255cf9647474d8bbb4141cbac67ba44 (table rebalancer_dynamic_table_new_name [id=e0549c7b124b4f83866eacba79670952]) successfully deleted
I20240502 21:27:18.369820 8906 ts_tablet_manager.cc:1888] T 5255cf9647474d8bbb4141cbac67ba44 P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:18.370005 8906 log.cc:1196] T 5255cf9647474d8bbb4141cbac67ba44 P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/5255cf9647474d8bbb4141cbac67ba44
I20240502 21:27:18.370396 8906 ts_tablet_manager.cc:1909] T 5255cf9647474d8bbb4141cbac67ba44 P 31a72286397c4edc9a63eca3412be168: Deleting consensus metadata
I20240502 21:27:18.371412 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet 5255cf9647474d8bbb4141cbac67ba44 (table rebalancer_dynamic_table_new_name [id=e0549c7b124b4f83866eacba79670952]) successfully deleted
I20240502 21:27:18.380892 8768 raft_consensus.cc:482] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:18.381090 8768 raft_consensus.cc:504] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:18.381907 8768 leader_election.cc:288] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:18.382334 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4cf59f48e874f42aed76d694246e7d3" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:27:18.382445 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4cf59f48e874f42aed76d694246e7d3" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:27:18.382581 4706 raft_consensus.cc:2454] T f4cf59f48e874f42aed76d694246e7d3 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:27:18.382690 4838 raft_consensus.cc:2454] T f4cf59f48e874f42aed76d694246e7d3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:27:18.382978 4511 leader_election.cc:302] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:27:18.383288 8768 raft_consensus.cc:2781] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:18.383385 8768 raft_consensus.cc:482] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:18.383509 8768 raft_consensus.cc:3037] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:18.385535 8768 raft_consensus.cc:504] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:18.386133 8768 leader_election.cc:288] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:18.386684 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4cf59f48e874f42aed76d694246e7d3" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:27:18.386926 4838 raft_consensus.cc:3037] T f4cf59f48e874f42aed76d694246e7d3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:18.387859 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4cf59f48e874f42aed76d694246e7d3" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:27:18.388082 4706 raft_consensus.cc:3037] T f4cf59f48e874f42aed76d694246e7d3 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:18.392334 4706 raft_consensus.cc:2454] T f4cf59f48e874f42aed76d694246e7d3 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:27:18.392990 4511 leader_election.cc:302] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:27:18.394680 4838 raft_consensus.cc:2454] T f4cf59f48e874f42aed76d694246e7d3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:27:18.395051 8768 raft_consensus.cc:2781] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:18.395445 8768 raft_consensus.cc:686] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:27:18.396049 8768 consensus_queue.cc:237] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:18.400485 4291 catalog_manager.cc:5484] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:18.451196 4291 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:18.458993 4422 tablet_service.cc:1467] Processing CreateTablet for tablet 1df176e7286d4b2b973d916444d246cf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fa32869925de476c8c42034a9aa065f5]), partition=
I20240502 21:27:18.459165 7463 tablet_service.cc:1467] Processing CreateTablet for tablet 1df176e7286d4b2b973d916444d246cf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fa32869925de476c8c42034a9aa065f5]), partition=
I20240502 21:27:18.459446 4818 tablet_service.cc:1467] Processing CreateTablet for tablet 1df176e7286d4b2b973d916444d246cf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fa32869925de476c8c42034a9aa065f5]), partition=
I20240502 21:27:18.459503 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1df176e7286d4b2b973d916444d246cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:18.459609 7463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1df176e7286d4b2b973d916444d246cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:18.459836 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1df176e7286d4b2b973d916444d246cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:18.466049 8883 tablet_bootstrap.cc:492] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:18.467516 8884 tablet_bootstrap.cc:492] T 1df176e7286d4b2b973d916444d246cf P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:18.467689 8883 tablet_bootstrap.cc:654] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:18.469628 8883 tablet_bootstrap.cc:492] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:18.469697 8884 tablet_bootstrap.cc:654] T 1df176e7286d4b2b973d916444d246cf P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:18.469810 8883 ts_tablet_manager.cc:1367] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 21:27:18.470604 8883 raft_consensus.cc:348] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:18.473623 8883 raft_consensus.cc:374] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:18.473786 8884 tablet_bootstrap.cc:492] T 1df176e7286d4b2b973d916444d246cf P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:18.473770 8883 raft_consensus.cc:729] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:18.473960 8884 ts_tablet_manager.cc:1367] T 1df176e7286d4b2b973d916444d246cf P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 21:27:18.474238 8883 consensus_queue.cc:260] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:18.474807 8884 raft_consensus.cc:348] T 1df176e7286d4b2b973d916444d246cf P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:18.475024 8884 raft_consensus.cc:374] T 1df176e7286d4b2b973d916444d246cf P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:18.475061 8883 ts_tablet_manager.cc:1398] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 21:27:18.475103 8884 raft_consensus.cc:729] T 1df176e7286d4b2b973d916444d246cf P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:18.475448 8909 tablet_bootstrap.cc:492] T 1df176e7286d4b2b973d916444d246cf P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:18.475348 8884 consensus_queue.cc:260] T 1df176e7286d4b2b973d916444d246cf P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:18.475919 8884 ts_tablet_manager.cc:1398] T 1df176e7286d4b2b973d916444d246cf P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:18.477222 8909 tablet_bootstrap.cc:654] T 1df176e7286d4b2b973d916444d246cf P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:18.482895 8909 tablet_bootstrap.cc:492] T 1df176e7286d4b2b973d916444d246cf P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:18.483084 8909 ts_tablet_manager.cc:1367] T 1df176e7286d4b2b973d916444d246cf P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240502 21:27:18.483909 8909 raft_consensus.cc:348] T 1df176e7286d4b2b973d916444d246cf P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:18.484104 8909 raft_consensus.cc:374] T 1df176e7286d4b2b973d916444d246cf P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:18.484174 8909 raft_consensus.cc:729] T 1df176e7286d4b2b973d916444d246cf P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:18.484409 8909 consensus_queue.cc:260] T 1df176e7286d4b2b973d916444d246cf P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:18.484951 8909 ts_tablet_manager.cc:1398] T 1df176e7286d4b2b973d916444d246cf P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:18.658618 8765 raft_consensus.cc:482] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:18.658905 8765 raft_consensus.cc:504] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:18.659541 8765 leader_election.cc:288] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:18.660001 7482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df176e7286d4b2b973d916444d246cf" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:18.660240 7482 raft_consensus.cc:2454] T 1df176e7286d4b2b973d916444d246cf P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:18.660241 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df176e7286d4b2b973d916444d246cf" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:27:18.660565 4839 raft_consensus.cc:2454] T 1df176e7286d4b2b973d916444d246cf P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:18.660733 4375 leader_election.cc:302] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:18.661067 8765 raft_consensus.cc:2781] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:18.661191 8765 raft_consensus.cc:482] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:18.661298 8765 raft_consensus.cc:3037] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:18.663643 8765 raft_consensus.cc:504] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:18.664158 8765 leader_election.cc:288] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:18.664578 7482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df176e7286d4b2b973d916444d246cf" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:18.664609 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df176e7286d4b2b973d916444d246cf" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:27:18.664819 7482 raft_consensus.cc:3037] T 1df176e7286d4b2b973d916444d246cf P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:18.664829 4839 raft_consensus.cc:3037] T 1df176e7286d4b2b973d916444d246cf P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:18.667706 4839 raft_consensus.cc:2454] T 1df176e7286d4b2b973d916444d246cf P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:18.667737 7482 raft_consensus.cc:2454] T 1df176e7286d4b2b973d916444d246cf P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:18.668130 4375 leader_election.cc:302] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:18.668650 8765 raft_consensus.cc:2781] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:18.668871 8765 raft_consensus.cc:686] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:27:18.669162 8765 consensus_queue.cc:237] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:18.673512 4291 catalog_manager.cc:5484] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:18.745028 8765 consensus_queue.cc:1035] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:18.752616 8892 consensus_queue.cc:1035] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:18.801599 4291 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:18.808470 7482 raft_consensus.cc:1261] T 1df176e7286d4b2b973d916444d246cf P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:18.809181 8765 consensus_queue.cc:1035] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:18.809413 4839 raft_consensus.cc:1261] T 1df176e7286d4b2b973d916444d246cf P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:18.810089 8765 consensus_queue.cc:1035] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:18.826591 4300 catalog_manager.cc:5576] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913) does not have the latest schema for tablet 1df176e7286d4b2b973d916444d246cf (table rebalancer_dynamic_table [id=fa32869925de476c8c42034a9aa065f5]). Expected version 1 got 0
I20240502 21:27:18.832437 8923 tablet.cc:1722] T 1df176e7286d4b2b973d916444d246cf P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:18.833598 8922 tablet.cc:1722] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:18.834298 8922 tablet.cc:1620] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:18.836820 4300 catalog_manager.cc:5576] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913) does not have the latest schema for tablet 1df176e7286d4b2b973d916444d246cf (table rebalancer_dynamic_table [id=fa32869925de476c8c42034a9aa065f5]). Expected version 1 got 0
I20240502 21:27:18.844303 4300 catalog_manager.cc:5576] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449) does not have the latest schema for tablet 1df176e7286d4b2b973d916444d246cf (table rebalancer_dynamic_table [id=fa32869925de476c8c42034a9aa065f5]). Expected version 1 got 0
I20240502 21:27:18.844797 8926 tablet.cc:1722] T 1df176e7286d4b2b973d916444d246cf P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:18.845679 8926 tablet.cc:1620] T 1df176e7286d4b2b973d916444d246cf P 73a1f2ca1f544de9b0e3b37015f85328: MemRowSet was empty: no flush needed.
I20240502 21:27:18.851455 8923 tablet.cc:1620] T 1df176e7286d4b2b973d916444d246cf P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:18.856880 4291 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=fa32869925de476c8c42034a9aa065f5] alter complete (version 1)
I20240502 21:27:18.867379 8902 consensus_queue.cc:1035] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:18.873103 8818 consensus_queue.cc:1035] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:18.895085 8768 consensus_queue.cc:1035] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:18.901037 8848 consensus_queue.cc:1035] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:18.919230 4291 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:18.928733 7328 tablet_service.cc:1467] Processing CreateTablet for tablet fb2cc722452f4d9798353d1bfae4bd4f (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=0953e644d4b143a9b5a77699e921c8de]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:18.928969 7463 tablet_service.cc:1467] Processing CreateTablet for tablet fb2cc722452f4d9798353d1bfae4bd4f (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=0953e644d4b143a9b5a77699e921c8de]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:18.929304 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb2cc722452f4d9798353d1bfae4bd4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:18.929220 4422 tablet_service.cc:1467] Processing CreateTablet for tablet fb2cc722452f4d9798353d1bfae4bd4f (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=0953e644d4b143a9b5a77699e921c8de]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:18.929416 7463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb2cc722452f4d9798353d1bfae4bd4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:18.929720 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb2cc722452f4d9798353d1bfae4bd4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:18.932253 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 56517ab7e5e447fea86868dd0a5b3c25 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=0953e644d4b143a9b5a77699e921c8de]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:18.932701 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56517ab7e5e447fea86868dd0a5b3c25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:18.936046 4555 tablet_service.cc:1467] Processing CreateTablet for tablet d6370cfa3d804e0fa5dc2a0f71d78840 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=0953e644d4b143a9b5a77699e921c8de]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:18.936196 8909 tablet_bootstrap.cc:492] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:18.936543 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6370cfa3d804e0fa5dc2a0f71d78840. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:18.936467 7462 tablet_service.cc:1467] Processing CreateTablet for tablet d6370cfa3d804e0fa5dc2a0f71d78840 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=0953e644d4b143a9b5a77699e921c8de]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:18.936887 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6370cfa3d804e0fa5dc2a0f71d78840. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:18.938536 8909 tablet_bootstrap.cc:654] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:18.940690 7195 tablet_service.cc:1467] Processing CreateTablet for tablet 56517ab7e5e447fea86868dd0a5b3c25 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=0953e644d4b143a9b5a77699e921c8de]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:18.941157 7195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56517ab7e5e447fea86868dd0a5b3c25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:18.941470 7595 tablet_service.cc:1467] Processing CreateTablet for tablet 56517ab7e5e447fea86868dd0a5b3c25 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=0953e644d4b143a9b5a77699e921c8de]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:18.941912 7595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56517ab7e5e447fea86868dd0a5b3c25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:18.945021 7329 tablet_service.cc:1467] Processing CreateTablet for tablet d6370cfa3d804e0fa5dc2a0f71d78840 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=0953e644d4b143a9b5a77699e921c8de]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:18.945452 7329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6370cfa3d804e0fa5dc2a0f71d78840. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:18.953929 8909 tablet_bootstrap.cc:492] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:18.954195 8909 ts_tablet_manager.cc:1367] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.000s
I20240502 21:27:18.955148 8909 raft_consensus.cc:348] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:18.955505 8909 raft_consensus.cc:374] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:18.955690 8909 raft_consensus.cc:729] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:18.956043 8909 consensus_queue.cc:260] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:18.956723 8909 ts_tablet_manager.cc:1398] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:18.958453 4291 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:18.959080 8909 tablet_bootstrap.cc:492] T d6370cfa3d804e0fa5dc2a0f71d78840 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:18.961032 8909 tablet_bootstrap.cc:654] T d6370cfa3d804e0fa5dc2a0f71d78840 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:18.973387 8909 tablet_bootstrap.cc:492] T d6370cfa3d804e0fa5dc2a0f71d78840 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:18.973747 8909 ts_tablet_manager.cc:1367] T d6370cfa3d804e0fa5dc2a0f71d78840 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.006s
I20240502 21:27:18.974897 8922 tablet.cc:1722] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:18.974817 8909 raft_consensus.cc:348] T d6370cfa3d804e0fa5dc2a0f71d78840 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:18.975380 8909 raft_consensus.cc:374] T d6370cfa3d804e0fa5dc2a0f71d78840 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:18.975441 8922 tablet.cc:1620] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:18.975616 8909 raft_consensus.cc:729] T d6370cfa3d804e0fa5dc2a0f71d78840 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:18.975939 8909 consensus_queue.cc:260] T d6370cfa3d804e0fa5dc2a0f71d78840 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:18.976825 8909 ts_tablet_manager.cc:1398] T d6370cfa3d804e0fa5dc2a0f71d78840 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:18.978705 8926 tablet.cc:1722] T 1df176e7286d4b2b973d916444d246cf P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:18.979192 8926 tablet.cc:1620] T 1df176e7286d4b2b973d916444d246cf P 73a1f2ca1f544de9b0e3b37015f85328: MemRowSet was empty: no flush needed.
I20240502 21:27:18.981424 8852 raft_consensus.cc:482] T d6370cfa3d804e0fa5dc2a0f71d78840 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:18.981575 8852 raft_consensus.cc:504] T d6370cfa3d804e0fa5dc2a0f71d78840 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:18.982532 8852 leader_election.cc:288] T d6370cfa3d804e0fa5dc2a0f71d78840 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:18.986290 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6370cfa3d804e0fa5dc2a0f71d78840" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:18.989559 4294 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=fa32869925de476c8c42034a9aa065f5] alter complete (version 2)
W20240502 21:27:18.990444 7419 leader_election.cc:341] T d6370cfa3d804e0fa5dc2a0f71d78840 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:18.994655 8923 tablet.cc:1722] T 1df176e7286d4b2b973d916444d246cf P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:18.995246 8923 tablet.cc:1620] T 1df176e7286d4b2b973d916444d246cf P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:19.003553 8938 tablet_bootstrap.cc:492] T 56517ab7e5e447fea86868dd0a5b3c25 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:19.004341 8937 tablet_bootstrap.cc:492] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:19.007653 8939 tablet_bootstrap.cc:492] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:19.011634 8938 tablet_bootstrap.cc:654] T 56517ab7e5e447fea86868dd0a5b3c25 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:19.013335 8937 tablet_bootstrap.cc:654] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:19.013816 8939 tablet_bootstrap.cc:654] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:19.016852 8942 tablet_bootstrap.cc:492] T 56517ab7e5e447fea86868dd0a5b3c25 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:19.020865 8942 tablet_bootstrap.cc:654] T 56517ab7e5e447fea86868dd0a5b3c25 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:19.021624 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6370cfa3d804e0fa5dc2a0f71d78840" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
W20240502 21:27:19.022274 7420 leader_election.cc:341] T d6370cfa3d804e0fa5dc2a0f71d78840 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:19.022452 7420 leader_election.cc:302] T d6370cfa3d804e0fa5dc2a0f71d78840 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a72286397c4edc9a63eca3412be168; no voters: 7a736a8ebb164ce0a051c0fccd76c3b8, d0456c5e55a542128688904e003d403c
I20240502 21:27:19.022863 8852 raft_consensus.cc:2726] T d6370cfa3d804e0fa5dc2a0f71d78840 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:27:19.025137 8939 tablet_bootstrap.cc:492] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:19.025749 8939 ts_tablet_manager.cc:1367] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.000s
I20240502 21:27:19.026961 8938 tablet_bootstrap.cc:492] T 56517ab7e5e447fea86868dd0a5b3c25 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:27:19.028630 8938 ts_tablet_manager.cc:1367] T 56517ab7e5e447fea86868dd0a5b3c25 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20240502 21:27:19.028609 8935 tablet_bootstrap.cc:492] T 56517ab7e5e447fea86868dd0a5b3c25 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:19.026574 8939 raft_consensus.cc:348] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:19.029049 8939 raft_consensus.cc:374] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:19.029402 8939 raft_consensus.cc:729] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:19.029546 8938 raft_consensus.cc:348] T 56517ab7e5e447fea86868dd0a5b3c25 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:19.029781 8938 raft_consensus.cc:374] T 56517ab7e5e447fea86868dd0a5b3c25 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:19.029886 8938 raft_consensus.cc:729] T 56517ab7e5e447fea86868dd0a5b3c25 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:19.029937 8939 consensus_queue.cc:260] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:19.030381 8938 consensus_queue.cc:260] T 56517ab7e5e447fea86868dd0a5b3c25 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:19.031033 8938 ts_tablet_manager.cc:1398] T 56517ab7e5e447fea86868dd0a5b3c25 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:19.031745 8935 tablet_bootstrap.cc:654] T 56517ab7e5e447fea86868dd0a5b3c25 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:19.032339 8937 tablet_bootstrap.cc:492] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:19.033258 8939 ts_tablet_manager.cc:1398] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 21:27:19.032585 8937 ts_tablet_manager.cc:1367] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.000s
I20240502 21:27:19.035681 8937 raft_consensus.cc:348] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:19.036522 8937 raft_consensus.cc:374] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:19.036724 8937 raft_consensus.cc:729] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:19.037092 8937 consensus_queue.cc:260] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:19.039430 8937 ts_tablet_manager.cc:1398] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:27:19.042122 8942 tablet_bootstrap.cc:492] T 56517ab7e5e447fea86868dd0a5b3c25 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:19.042322 8942 ts_tablet_manager.cc:1367] T 56517ab7e5e447fea86868dd0a5b3c25 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20240502 21:27:19.043102 8942 raft_consensus.cc:348] T 56517ab7e5e447fea86868dd0a5b3c25 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:19.043300 8942 raft_consensus.cc:374] T 56517ab7e5e447fea86868dd0a5b3c25 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:19.043399 8942 raft_consensus.cc:729] T 56517ab7e5e447fea86868dd0a5b3c25 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:19.043651 8942 consensus_queue.cc:260] T 56517ab7e5e447fea86868dd0a5b3c25 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:19.043869 8933 tablet_bootstrap.cc:492] T fb2cc722452f4d9798353d1bfae4bd4f P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:19.044363 8942 ts_tablet_manager.cc:1398] T 56517ab7e5e447fea86868dd0a5b3c25 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:19.051524 8933 tablet_bootstrap.cc:654] T fb2cc722452f4d9798353d1bfae4bd4f P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:19.052538 8935 tablet_bootstrap.cc:492] T 56517ab7e5e447fea86868dd0a5b3c25 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:19.052768 8935 ts_tablet_manager.cc:1367] T 56517ab7e5e447fea86868dd0a5b3c25 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.003s
I20240502 21:27:19.053711 8933 tablet_bootstrap.cc:492] T fb2cc722452f4d9798353d1bfae4bd4f P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:19.053532 8935 raft_consensus.cc:348] T 56517ab7e5e447fea86868dd0a5b3c25 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:19.053934 8933 ts_tablet_manager.cc:1367] T fb2cc722452f4d9798353d1bfae4bd4f P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240502 21:27:19.053797 8935 raft_consensus.cc:374] T 56517ab7e5e447fea86868dd0a5b3c25 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:19.054159 8935 raft_consensus.cc:729] T 56517ab7e5e447fea86868dd0a5b3c25 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:19.054468 8935 consensus_queue.cc:260] T 56517ab7e5e447fea86868dd0a5b3c25 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:19.054844 8933 raft_consensus.cc:348] T fb2cc722452f4d9798353d1bfae4bd4f P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:19.055064 8933 raft_consensus.cc:374] T fb2cc722452f4d9798353d1bfae4bd4f P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:19.055169 8933 raft_consensus.cc:729] T fb2cc722452f4d9798353d1bfae4bd4f P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:19.055292 8935 ts_tablet_manager.cc:1398] T 56517ab7e5e447fea86868dd0a5b3c25 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:19.055406 8933 consensus_queue.cc:260] T fb2cc722452f4d9798353d1bfae4bd4f P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:19.056239 8933 ts_tablet_manager.cc:1398] T fb2cc722452f4d9798353d1bfae4bd4f P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:19.056577 8933 tablet_bootstrap.cc:492] T d6370cfa3d804e0fa5dc2a0f71d78840 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:19.064976 8651 raft_consensus.cc:482] T 56517ab7e5e447fea86868dd0a5b3c25 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:19.065201 8651 raft_consensus.cc:504] T 56517ab7e5e447fea86868dd0a5b3c25 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:19.065933 8651 leader_election.cc:288] T 56517ab7e5e447fea86868dd0a5b3c25 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:19.066556 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56517ab7e5e447fea86868dd0a5b3c25" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:27:19.066816 4705 raft_consensus.cc:2454] T 56517ab7e5e447fea86868dd0a5b3c25 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:19.067195 7616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56517ab7e5e447fea86868dd0a5b3c25" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:19.067516 7616 raft_consensus.cc:2454] T 56517ab7e5e447fea86868dd0a5b3c25 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:19.067972 7151 leader_election.cc:302] T 56517ab7e5e447fea86868dd0a5b3c25 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:19.068245 8651 raft_consensus.cc:2781] T 56517ab7e5e447fea86868dd0a5b3c25 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:19.068342 8651 raft_consensus.cc:482] T 56517ab7e5e447fea86868dd0a5b3c25 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:19.068441 8651 raft_consensus.cc:3037] T 56517ab7e5e447fea86868dd0a5b3c25 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:19.071491 8651 raft_consensus.cc:504] T 56517ab7e5e447fea86868dd0a5b3c25 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:19.072094 8933 tablet_bootstrap.cc:654] T d6370cfa3d804e0fa5dc2a0f71d78840 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:19.072764 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56517ab7e5e447fea86868dd0a5b3c25" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:27:19.073007 4705 raft_consensus.cc:3037] T 56517ab7e5e447fea86868dd0a5b3c25 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:19.074880 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56517ab7e5e447fea86868dd0a5b3c25" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:19.075129 7618 raft_consensus.cc:3037] T 56517ab7e5e447fea86868dd0a5b3c25 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:19.075845 4705 raft_consensus.cc:2454] T 56517ab7e5e447fea86868dd0a5b3c25 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:19.078289 7618 raft_consensus.cc:2454] T 56517ab7e5e447fea86868dd0a5b3c25 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:19.079449 7151 leader_election.cc:302] T 56517ab7e5e447fea86868dd0a5b3c25 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:19.079814 8875 raft_consensus.cc:2781] T 56517ab7e5e447fea86868dd0a5b3c25 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:19.082160 8651 leader_election.cc:288] T 56517ab7e5e447fea86868dd0a5b3c25 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:19.083041 8875 raft_consensus.cc:686] T 56517ab7e5e447fea86868dd0a5b3c25 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:27:19.083370 8875 consensus_queue.cc:237] T 56517ab7e5e447fea86868dd0a5b3c25 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:19.088053 4294 catalog_manager.cc:5484] T 56517ab7e5e447fea86868dd0a5b3c25 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69). New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:19.089744 8933 tablet_bootstrap.cc:492] T d6370cfa3d804e0fa5dc2a0f71d78840 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:19.089951 8933 ts_tablet_manager.cc:1367] T d6370cfa3d804e0fa5dc2a0f71d78840 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.002s
I20240502 21:27:19.090986 8933 raft_consensus.cc:348] T d6370cfa3d804e0fa5dc2a0f71d78840 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:19.091284 8933 raft_consensus.cc:374] T d6370cfa3d804e0fa5dc2a0f71d78840 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:19.091542 8933 raft_consensus.cc:729] T d6370cfa3d804e0fa5dc2a0f71d78840 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:19.091904 8933 consensus_queue.cc:260] T d6370cfa3d804e0fa5dc2a0f71d78840 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:19.092370 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:19.092764 8933 ts_tablet_manager.cc:1398] T d6370cfa3d804e0fa5dc2a0f71d78840 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:19.102386 8922 tablet.cc:1722] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:19.103822 8926 tablet.cc:1722] T 1df176e7286d4b2b973d916444d246cf P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:19.104252 8931 raft_consensus.cc:482] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:19.104413 8931 raft_consensus.cc:504] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:19.105326 8931 leader_election.cc:288] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:19.106086 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6370cfa3d804e0fa5dc2a0f71d78840" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:19.106307 7349 raft_consensus.cc:2454] T d6370cfa3d804e0fa5dc2a0f71d78840 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:27:19.106746 4510 leader_election.cc:302] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a736a8ebb164ce0a051c0fccd76c3b8, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:19.107127 8931 raft_consensus.cc:2781] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:19.107266 8931 raft_consensus.cc:482] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:19.107379 8931 raft_consensus.cc:3037] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:19.112041 7482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6370cfa3d804e0fa5dc2a0f71d78840" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:19.112299 7482 raft_consensus.cc:2454] T d6370cfa3d804e0fa5dc2a0f71d78840 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:27:19.113034 8765 raft_consensus.cc:482] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:19.113211 8765 raft_consensus.cc:504] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:19.114058 8765 leader_election.cc:288] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:19.114707 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb2cc722452f4d9798353d1bfae4bd4f" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:19.114696 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb2cc722452f4d9798353d1bfae4bd4f" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:19.114923 7349 raft_consensus.cc:2454] T fb2cc722452f4d9798353d1bfae4bd4f P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:19.114951 7484 raft_consensus.cc:2454] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:19.114881 8923 tablet.cc:1722] T 1df176e7286d4b2b973d916444d246cf P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:19.115339 4377 leader_election.cc:302] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:19.115687 8892 raft_consensus.cc:2781] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:19.115819 8892 raft_consensus.cc:482] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:19.115927 8892 raft_consensus.cc:3037] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:19.110101 8931 raft_consensus.cc:504] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:19.116179 4291 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=fa32869925de476c8c42034a9aa065f5] alter complete (version 3)
I20240502 21:27:19.117167 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6370cfa3d804e0fa5dc2a0f71d78840" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:19.117381 7349 raft_consensus.cc:3037] T d6370cfa3d804e0fa5dc2a0f71d78840 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:19.118551 8892 raft_consensus.cc:504] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:19.119174 8892 leader_election.cc:288] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:19.119637 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb2cc722452f4d9798353d1bfae4bd4f" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:19.119860 7350 raft_consensus.cc:3037] T fb2cc722452f4d9798353d1bfae4bd4f P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:19.119886 8931 leader_election.cc:288] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Requested vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:19.120101 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb2cc722452f4d9798353d1bfae4bd4f" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:19.120255 7349 raft_consensus.cc:2454] T d6370cfa3d804e0fa5dc2a0f71d78840 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:27:19.120322 7484 raft_consensus.cc:3037] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:19.120677 4510 leader_election.cc:302] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a736a8ebb164ce0a051c0fccd76c3b8, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:19.121033 8931 raft_consensus.cc:2781] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:19.121243 8931 raft_consensus.cc:686] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:27:19.121603 8931 consensus_queue.cc:237] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:19.122397 7350 raft_consensus.cc:2454] T fb2cc722452f4d9798353d1bfae4bd4f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:19.122951 7484 raft_consensus.cc:2454] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:19.123432 4375 leader_election.cc:302] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:19.123903 8892 raft_consensus.cc:2781] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:19.124058 8892 raft_consensus.cc:686] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:27:19.124403 8892 consensus_queue.cc:237] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:19.126452 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6370cfa3d804e0fa5dc2a0f71d78840" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:19.126711 7484 raft_consensus.cc:3037] T d6370cfa3d804e0fa5dc2a0f71d78840 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:19.130369 7484 raft_consensus.cc:2454] T d6370cfa3d804e0fa5dc2a0f71d78840 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:27:19.131526 4291 catalog_manager.cc:5484] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:19.134397 4294 catalog_manager.cc:5484] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:19.229614 4294 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:19.229883 4294 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:19.233963 4294 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 1df176e7286d4b2b973d916444d246cf
I20240502 21:27:19.234793 4422 tablet_service.cc:1514] Processing DeleteTablet for tablet 1df176e7286d4b2b973d916444d246cf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:19 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:19.235795 8956 tablet_replica.cc:330] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:19.236136 8956 raft_consensus.cc:2229] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:19.236577 8956 raft_consensus.cc:2258] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:19.236903 7462 tablet_service.cc:1514] Processing DeleteTablet for tablet 1df176e7286d4b2b973d916444d246cf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:19 UTC) from {username='slave'} at 127.0.0.1:37182
I20240502 21:27:19.237605 4818 tablet_service.cc:1514] Processing DeleteTablet for tablet 1df176e7286d4b2b973d916444d246cf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:19 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:27:19.238407 8956 ts_tablet_manager.cc:1875] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:19.238821 8958 tablet_replica.cc:330] T 1df176e7286d4b2b973d916444d246cf P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:27:19.239071 8958 raft_consensus.cc:2229] T 1df176e7286d4b2b973d916444d246cf P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:19.239122 8957 tablet_replica.cc:330] T 1df176e7286d4b2b973d916444d246cf P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:27:19.239357 8958 raft_consensus.cc:2258] T 1df176e7286d4b2b973d916444d246cf P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:19.240221 8958 ts_tablet_manager.cc:1875] T 1df176e7286d4b2b973d916444d246cf P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:19.239410 8957 raft_consensus.cc:2229] T 1df176e7286d4b2b973d916444d246cf P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:19.240921 8957 raft_consensus.cc:2258] T 1df176e7286d4b2b973d916444d246cf P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:19.244150 8957 ts_tablet_manager.cc:1875] T 1df176e7286d4b2b973d916444d246cf P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:19.246858 8958 ts_tablet_manager.cc:1888] T 1df176e7286d4b2b973d916444d246cf P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:19.247043 8958 log.cc:1196] T 1df176e7286d4b2b973d916444d246cf P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/1df176e7286d4b2b973d916444d246cf
I20240502 21:27:19.247634 8958 ts_tablet_manager.cc:1909] T 1df176e7286d4b2b973d916444d246cf P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:27:19.247857 8956 ts_tablet_manager.cc:1888] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:19.248049 8956 log.cc:1196] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/1df176e7286d4b2b973d916444d246cf
I20240502 21:27:19.248587 8956 ts_tablet_manager.cc:1909] T 1df176e7286d4b2b973d916444d246cf P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:19.248869 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet 1df176e7286d4b2b973d916444d246cf (table rebalancer_dynamic_table_new_name [id=fa32869925de476c8c42034a9aa065f5]) successfully deleted
I20240502 21:27:19.249652 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet 1df176e7286d4b2b973d916444d246cf (table rebalancer_dynamic_table_new_name [id=fa32869925de476c8c42034a9aa065f5]) successfully deleted
I20240502 21:27:19.250852 8957 ts_tablet_manager.cc:1888] T 1df176e7286d4b2b973d916444d246cf P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:19.250978 8957 log.cc:1196] T 1df176e7286d4b2b973d916444d246cf P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/1df176e7286d4b2b973d916444d246cf
I20240502 21:27:19.251384 8957 ts_tablet_manager.cc:1909] T 1df176e7286d4b2b973d916444d246cf P 31a72286397c4edc9a63eca3412be168: Deleting consensus metadata
I20240502 21:27:19.252390 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet 1df176e7286d4b2b973d916444d246cf (table rebalancer_dynamic_table_new_name [id=fa32869925de476c8c42034a9aa065f5]) successfully deleted
I20240502 21:27:19.340330 4294 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:19.348002 4422 tablet_service.cc:1467] Processing CreateTablet for tablet c29d52114111471ca87a5f41f493bbb5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5665c045531041388f6095c33442c337]), partition=
I20240502 21:27:19.348198 4818 tablet_service.cc:1467] Processing CreateTablet for tablet c29d52114111471ca87a5f41f493bbb5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5665c045531041388f6095c33442c337]), partition=
I20240502 21:27:19.348479 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c29d52114111471ca87a5f41f493bbb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:19.348677 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c29d52114111471ca87a5f41f493bbb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:19.353672 8939 tablet_bootstrap.cc:492] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:19.359596 8939 tablet_bootstrap.cc:654] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:19.361634 8939 tablet_bootstrap.cc:492] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:19.361810 8939 ts_tablet_manager.cc:1367] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240502 21:27:19.362644 8939 raft_consensus.cc:348] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:19.362880 8939 raft_consensus.cc:374] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:19.362979 8939 raft_consensus.cc:729] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:19.363255 8939 consensus_queue.cc:260] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:19.364221 8939 ts_tablet_manager.cc:1398] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:19.364584 8959 tablet_bootstrap.cc:492] T c29d52114111471ca87a5f41f493bbb5 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:19.364580 7462 tablet_service.cc:1467] Processing CreateTablet for tablet c29d52114111471ca87a5f41f493bbb5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5665c045531041388f6095c33442c337]), partition=
I20240502 21:27:19.364984 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c29d52114111471ca87a5f41f493bbb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:19.366356 8959 tablet_bootstrap.cc:654] T c29d52114111471ca87a5f41f493bbb5 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:19.371884 8909 tablet_bootstrap.cc:492] T c29d52114111471ca87a5f41f493bbb5 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:19.373482 8909 tablet_bootstrap.cc:654] T c29d52114111471ca87a5f41f493bbb5 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:19.388222 8909 tablet_bootstrap.cc:492] T c29d52114111471ca87a5f41f493bbb5 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:19.388437 8909 ts_tablet_manager.cc:1367] T c29d52114111471ca87a5f41f493bbb5 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.006s
I20240502 21:27:19.389277 8909 raft_consensus.cc:348] T c29d52114111471ca87a5f41f493bbb5 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:19.389566 8909 raft_consensus.cc:374] T c29d52114111471ca87a5f41f493bbb5 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:19.389649 8909 raft_consensus.cc:729] T c29d52114111471ca87a5f41f493bbb5 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:19.389909 8909 consensus_queue.cc:260] T c29d52114111471ca87a5f41f493bbb5 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:19.390507 8909 ts_tablet_manager.cc:1398] T c29d52114111471ca87a5f41f493bbb5 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:19.401507 8959 tablet_bootstrap.cc:492] T c29d52114111471ca87a5f41f493bbb5 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:19.401783 8959 ts_tablet_manager.cc:1367] T c29d52114111471ca87a5f41f493bbb5 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.029s
I20240502 21:27:19.402743 8959 raft_consensus.cc:348] T c29d52114111471ca87a5f41f493bbb5 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:19.403295 8959 raft_consensus.cc:374] T c29d52114111471ca87a5f41f493bbb5 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:19.403455 8959 raft_consensus.cc:729] T c29d52114111471ca87a5f41f493bbb5 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:19.403759 8959 consensus_queue.cc:260] T c29d52114111471ca87a5f41f493bbb5 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:19.404786 8959 ts_tablet_manager.cc:1398] T c29d52114111471ca87a5f41f493bbb5 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:19.482106 8875 consensus_queue.cc:1035] T 56517ab7e5e447fea86868dd0a5b3c25 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:19.488075 8952 consensus_queue.cc:1035] T 56517ab7e5e447fea86868dd0a5b3c25 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:19.521874 8931 consensus_queue.cc:1035] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:19.527161 8931 consensus_queue.cc:1035] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:19.535724 8765 consensus_queue.cc:1035] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:19.537497 8765 consensus_queue.cc:1035] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:19.638681 8765 raft_consensus.cc:482] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:19.638865 8765 raft_consensus.cc:504] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:19.639540 8765 leader_election.cc:288] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:19.639968 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c29d52114111471ca87a5f41f493bbb5" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:27:19.640086 7482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c29d52114111471ca87a5f41f493bbb5" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:19.640254 4839 raft_consensus.cc:2454] T c29d52114111471ca87a5f41f493bbb5 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:19.640319 7482 raft_consensus.cc:2454] T c29d52114111471ca87a5f41f493bbb5 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:19.640770 4375 leader_election.cc:302] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:19.641088 8765 raft_consensus.cc:2781] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:19.641227 8765 raft_consensus.cc:482] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:19.641305 8765 raft_consensus.cc:3037] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:19.643965 4291 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:19.644984 8765 raft_consensus.cc:504] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:19.645725 8765 leader_election.cc:288] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:19.646185 7482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c29d52114111471ca87a5f41f493bbb5" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:19.646375 7482 raft_consensus.cc:3037] T c29d52114111471ca87a5f41f493bbb5 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:19.646245 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c29d52114111471ca87a5f41f493bbb5" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:27:19.647265 4839 raft_consensus.cc:3037] T c29d52114111471ca87a5f41f493bbb5 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:19.648919 7482 raft_consensus.cc:2454] T c29d52114111471ca87a5f41f493bbb5 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:19.649334 4375 leader_election.cc:302] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:19.649662 8765 raft_consensus.cc:2781] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:19.649775 8765 raft_consensus.cc:686] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:27:19.650164 4839 raft_consensus.cc:2454] T c29d52114111471ca87a5f41f493bbb5 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:19.650105 8765 consensus_queue.cc:237] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:19.656637 4291 catalog_manager.cc:5484] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:27:19.657366 7462 tablet_service.cc:1467] Processing CreateTablet for tablet ebcf1894471f4fb7b951a730c2a781dd (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8781896475b940309fa802266d577ff0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:19.657756 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebcf1894471f4fb7b951a730c2a781dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:19.657999 4555 tablet_service.cc:1467] Processing CreateTablet for tablet ebcf1894471f4fb7b951a730c2a781dd (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8781896475b940309fa802266d577ff0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:19.658440 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebcf1894471f4fb7b951a730c2a781dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:19.661614 7595 tablet_service.cc:1467] Processing CreateTablet for tablet 612a0397b1754fda81c4a0019067a0d3 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8781896475b940309fa802266d577ff0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:19.662046 7595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 612a0397b1754fda81c4a0019067a0d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:19.662425 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 831d24800da54da9b22f9e1bca474755 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8781896475b940309fa802266d577ff0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:19.662788 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 831d24800da54da9b22f9e1bca474755. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:19.663267 8909 tablet_bootstrap.cc:492] T ebcf1894471f4fb7b951a730c2a781dd P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:19.664747 4422 tablet_service.cc:1467] Processing CreateTablet for tablet ebcf1894471f4fb7b951a730c2a781dd (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8781896475b940309fa802266d577ff0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:19.664902 8909 tablet_bootstrap.cc:654] T ebcf1894471f4fb7b951a730c2a781dd P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:19.665161 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebcf1894471f4fb7b951a730c2a781dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:19.666169 4818 tablet_service.cc:1467] Processing CreateTablet for tablet 831d24800da54da9b22f9e1bca474755 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8781896475b940309fa802266d577ff0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:19.666594 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 831d24800da54da9b22f9e1bca474755. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:19.670353 4421 tablet_service.cc:1467] Processing CreateTablet for tablet 612a0397b1754fda81c4a0019067a0d3 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8781896475b940309fa802266d577ff0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:19.670852 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 612a0397b1754fda81c4a0019067a0d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:19.674391 7195 tablet_service.cc:1467] Processing CreateTablet for tablet 831d24800da54da9b22f9e1bca474755 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8781896475b940309fa802266d577ff0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:19.674918 7195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 831d24800da54da9b22f9e1bca474755. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:19.675060 8980 tablet_bootstrap.cc:492] T 612a0397b1754fda81c4a0019067a0d3 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:19.676995 8980 tablet_bootstrap.cc:654] T 612a0397b1754fda81c4a0019067a0d3 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:19.680028 7194 tablet_service.cc:1467] Processing CreateTablet for tablet 612a0397b1754fda81c4a0019067a0d3 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8781896475b940309fa802266d577ff0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:19.680522 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 612a0397b1754fda81c4a0019067a0d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:19.684623 8939 tablet_bootstrap.cc:492] T 612a0397b1754fda81c4a0019067a0d3 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:19.686640 8939 tablet_bootstrap.cc:654] T 612a0397b1754fda81c4a0019067a0d3 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:19.694197 8909 tablet_bootstrap.cc:492] T ebcf1894471f4fb7b951a730c2a781dd P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:19.694391 8909 ts_tablet_manager.cc:1367] T ebcf1894471f4fb7b951a730c2a781dd P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20240502 21:27:19.697274 8959 tablet_bootstrap.cc:492] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:19.698973 8909 raft_consensus.cc:348] T ebcf1894471f4fb7b951a730c2a781dd P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:19.699215 8909 raft_consensus.cc:374] T ebcf1894471f4fb7b951a730c2a781dd P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:19.699316 8909 raft_consensus.cc:729] T ebcf1894471f4fb7b951a730c2a781dd P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:19.699595 8959 tablet_bootstrap.cc:654] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:19.699600 8909 consensus_queue.cc:260] T ebcf1894471f4fb7b951a730c2a781dd P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:19.702211 8982 tablet_bootstrap.cc:492] T ebcf1894471f4fb7b951a730c2a781dd P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:19.703539 8909 ts_tablet_manager.cc:1398] T ebcf1894471f4fb7b951a730c2a781dd P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240502 21:27:19.707784 8981 tablet_bootstrap.cc:492] T 831d24800da54da9b22f9e1bca474755 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:19.709951 8959 tablet_bootstrap.cc:492] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:19.710201 8959 ts_tablet_manager.cc:1367] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.001s
I20240502 21:27:19.710706 8939 tablet_bootstrap.cc:492] T 612a0397b1754fda81c4a0019067a0d3 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:19.710970 8939 ts_tablet_manager.cc:1367] T 612a0397b1754fda81c4a0019067a0d3 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.000s
I20240502 21:27:19.711169 8959 raft_consensus.cc:348] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:19.711474 8959 raft_consensus.cc:374] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:19.711647 8959 raft_consensus.cc:729] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:19.711988 8959 consensus_queue.cc:260] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:19.712306 8939 raft_consensus.cc:348] T 612a0397b1754fda81c4a0019067a0d3 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:19.712695 8939 raft_consensus.cc:374] T 612a0397b1754fda81c4a0019067a0d3 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:19.712881 8939 raft_consensus.cc:729] T 612a0397b1754fda81c4a0019067a0d3 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:19.712929 8959 ts_tablet_manager.cc:1398] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:19.713232 8939 consensus_queue.cc:260] T 612a0397b1754fda81c4a0019067a0d3 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:19.713925 8939 ts_tablet_manager.cc:1398] T 612a0397b1754fda81c4a0019067a0d3 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:19.714517 8981 tablet_bootstrap.cc:654] T 831d24800da54da9b22f9e1bca474755 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:19.714848 8939 tablet_bootstrap.cc:492] T ebcf1894471f4fb7b951a730c2a781dd P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:19.716491 8939 tablet_bootstrap.cc:654] T ebcf1894471f4fb7b951a730c2a781dd P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:19.714545 8986 tablet_bootstrap.cc:492] T 612a0397b1754fda81c4a0019067a0d3 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:19.717976 8981 tablet_bootstrap.cc:492] T 831d24800da54da9b22f9e1bca474755 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:27:19.718178 8981 ts_tablet_manager.cc:1367] T 831d24800da54da9b22f9e1bca474755 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240502 21:27:19.719045 8981 raft_consensus.cc:348] T 831d24800da54da9b22f9e1bca474755 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:19.719316 8981 raft_consensus.cc:374] T 831d24800da54da9b22f9e1bca474755 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:19.719463 8981 raft_consensus.cc:729] T 831d24800da54da9b22f9e1bca474755 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:19.719775 8981 consensus_queue.cc:260] T 831d24800da54da9b22f9e1bca474755 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:19.720417 8981 ts_tablet_manager.cc:1398] T 831d24800da54da9b22f9e1bca474755 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:19.720521 8986 tablet_bootstrap.cc:654] T 612a0397b1754fda81c4a0019067a0d3 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:19.719255 8982 tablet_bootstrap.cc:654] T ebcf1894471f4fb7b951a730c2a781dd P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:19.723093 8986 tablet_bootstrap.cc:492] T 612a0397b1754fda81c4a0019067a0d3 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:19.723778 8982 tablet_bootstrap.cc:492] T ebcf1894471f4fb7b951a730c2a781dd P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:19.723932 8982 ts_tablet_manager.cc:1367] T ebcf1894471f4fb7b951a730c2a781dd P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20240502 21:27:19.724628 8982 raft_consensus.cc:348] T ebcf1894471f4fb7b951a730c2a781dd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:19.724834 8982 raft_consensus.cc:374] T ebcf1894471f4fb7b951a730c2a781dd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:19.724944 8982 raft_consensus.cc:729] T ebcf1894471f4fb7b951a730c2a781dd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:19.725184 8982 consensus_queue.cc:260] T ebcf1894471f4fb7b951a730c2a781dd P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:19.725477 8986 ts_tablet_manager.cc:1367] T 612a0397b1754fda81c4a0019067a0d3 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20240502 21:27:19.726075 8986 raft_consensus.cc:348] T 612a0397b1754fda81c4a0019067a0d3 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:19.726255 8986 raft_consensus.cc:374] T 612a0397b1754fda81c4a0019067a0d3 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:19.726359 8986 raft_consensus.cc:729] T 612a0397b1754fda81c4a0019067a0d3 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:19.726599 8986 consensus_queue.cc:260] T 612a0397b1754fda81c4a0019067a0d3 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:19.727097 8986 ts_tablet_manager.cc:1398] T 612a0397b1754fda81c4a0019067a0d3 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:27:19.727372 8986 tablet_bootstrap.cc:492] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:19.728168 8980 tablet_bootstrap.cc:492] T 612a0397b1754fda81c4a0019067a0d3 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:19.728355 8980 ts_tablet_manager.cc:1367] T 612a0397b1754fda81c4a0019067a0d3 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.053s user 0.009s sys 0.010s
I20240502 21:27:19.729121 8986 tablet_bootstrap.cc:654] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:19.729231 8980 raft_consensus.cc:348] T 612a0397b1754fda81c4a0019067a0d3 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:19.729450 8980 raft_consensus.cc:374] T 612a0397b1754fda81c4a0019067a0d3 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:19.729557 8980 raft_consensus.cc:729] T 612a0397b1754fda81c4a0019067a0d3 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:19.729809 8980 consensus_queue.cc:260] T 612a0397b1754fda81c4a0019067a0d3 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:19.730376 8980 ts_tablet_manager.cc:1398] T 612a0397b1754fda81c4a0019067a0d3 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:19.736039 8982 ts_tablet_manager.cc:1398] T ebcf1894471f4fb7b951a730c2a781dd P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 21:27:19.737138 8939 tablet_bootstrap.cc:492] T ebcf1894471f4fb7b951a730c2a781dd P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:19.737327 8939 ts_tablet_manager.cc:1367] T ebcf1894471f4fb7b951a730c2a781dd P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.003s
I20240502 21:27:19.737967 8939 raft_consensus.cc:348] T ebcf1894471f4fb7b951a730c2a781dd P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:19.738204 8939 raft_consensus.cc:374] T ebcf1894471f4fb7b951a730c2a781dd P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:19.738315 8939 raft_consensus.cc:729] T ebcf1894471f4fb7b951a730c2a781dd P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:19.738548 8939 consensus_queue.cc:260] T ebcf1894471f4fb7b951a730c2a781dd P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:19.739121 8939 ts_tablet_manager.cc:1398] T ebcf1894471f4fb7b951a730c2a781dd P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:19.739773 8986 tablet_bootstrap.cc:492] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:19.739940 8986 ts_tablet_manager.cc:1367] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240502 21:27:19.740664 8986 raft_consensus.cc:348] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:19.740931 8986 raft_consensus.cc:374] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:19.741025 8986 raft_consensus.cc:729] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:19.741277 8986 consensus_queue.cc:260] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:19.741797 8986 ts_tablet_manager.cc:1398] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:27:19.782847 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:19.788938 7482 raft_consensus.cc:1261] T c29d52114111471ca87a5f41f493bbb5 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:19.789584 4839 raft_consensus.cc:1261] T c29d52114111471ca87a5f41f493bbb5 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:19.789676 8765 consensus_queue.cc:1035] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:19.790215 8892 consensus_queue.cc:1035] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:19.797461 4300 catalog_manager.cc:5576] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913) does not have the latest schema for tablet c29d52114111471ca87a5f41f493bbb5 (table rebalancer_dynamic_table [id=5665c045531041388f6095c33442c337]). Expected version 1 got 0
I20240502 21:27:19.799427 4294 catalog_manager.cc:5576] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449) does not have the latest schema for tablet c29d52114111471ca87a5f41f493bbb5 (table rebalancer_dynamic_table [id=5665c045531041388f6095c33442c337]). Expected version 1 got 0
I20240502 21:27:19.804530 8998 tablet.cc:1722] T c29d52114111471ca87a5f41f493bbb5 P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:19.805568 8998 tablet.cc:1620] T c29d52114111471ca87a5f41f493bbb5 P 73a1f2ca1f544de9b0e3b37015f85328: MemRowSet was empty: no flush needed.
I20240502 21:27:19.819015 4294 catalog_manager.cc:5576] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449) does not have the latest schema for tablet c29d52114111471ca87a5f41f493bbb5 (table rebalancer_dynamic_table [id=5665c045531041388f6095c33442c337]). Expected version 1 got 0
I20240502 21:27:19.827163 4294 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=5665c045531041388f6095c33442c337] alter complete (version 1)
I20240502 21:27:19.833457 8952 raft_consensus.cc:482] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:19.833678 8952 raft_consensus.cc:504] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:19.834475 8952 leader_election.cc:288] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:19.834898 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831d24800da54da9b22f9e1bca474755" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:27:19.835187 4705 raft_consensus.cc:2454] T 831d24800da54da9b22f9e1bca474755 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:19.835740 7151 leader_election.cc:302] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:19.836383 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831d24800da54da9b22f9e1bca474755" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:27:19.836680 8952 raft_consensus.cc:2781] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:19.836647 4839 raft_consensus.cc:2454] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:19.836771 8952 raft_consensus.cc:482] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:19.836881 8952 raft_consensus.cc:3037] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:19.840119 8952 raft_consensus.cc:504] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:19.842203 8952 leader_election.cc:288] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:19.842128 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831d24800da54da9b22f9e1bca474755" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:27:19.842362 4705 raft_consensus.cc:3037] T 831d24800da54da9b22f9e1bca474755 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:19.843789 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831d24800da54da9b22f9e1bca474755" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:27:19.844013 4839 raft_consensus.cc:3037] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:19.845515 4705 raft_consensus.cc:2454] T 831d24800da54da9b22f9e1bca474755 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:19.846340 7151 leader_election.cc:302] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:19.846768 4839 raft_consensus.cc:2454] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:19.846975 8999 tablet.cc:1722] T c29d52114111471ca87a5f41f493bbb5 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:19.849630 8999 tablet.cc:1620] T c29d52114111471ca87a5f41f493bbb5 P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:19.850111 4294 catalog_manager.cc:5576] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913) does not have the latest schema for tablet c29d52114111471ca87a5f41f493bbb5 (table rebalancer_dynamic_table [id=5665c045531041388f6095c33442c337]). Expected version 1 got 0
I20240502 21:27:19.850579 8875 raft_consensus.cc:2781] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:19.850620 8997 tablet.cc:1722] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:19.851123 8997 tablet.cc:1620] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:19.857833 8875 raft_consensus.cc:686] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:27:19.858144 8875 consensus_queue.cc:237] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:19.870438 4294 catalog_manager.cc:5484] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69). New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:19.893908 8931 raft_consensus.cc:482] T ebcf1894471f4fb7b951a730c2a781dd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:19.894122 8931 raft_consensus.cc:504] T ebcf1894471f4fb7b951a730c2a781dd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:19.895041 8931 leader_election.cc:288] T ebcf1894471f4fb7b951a730c2a781dd P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:19.895614 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcf1894471f4fb7b951a730c2a781dd" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:27:19.895893 4440 raft_consensus.cc:2454] T ebcf1894471f4fb7b951a730c2a781dd P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:27:19.896746 4508 leader_election.cc:302] T ebcf1894471f4fb7b951a730c2a781dd P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:27:19.897464 8931 raft_consensus.cc:2781] T ebcf1894471f4fb7b951a730c2a781dd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:19.897603 8931 raft_consensus.cc:482] T ebcf1894471f4fb7b951a730c2a781dd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:19.897722 8931 raft_consensus.cc:3037] T ebcf1894471f4fb7b951a730c2a781dd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:19.900409 8931 raft_consensus.cc:504] T ebcf1894471f4fb7b951a730c2a781dd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:19.901439 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcf1894471f4fb7b951a730c2a781dd" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:27:19.901439 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcf1894471f4fb7b951a730c2a781dd" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:19.901665 7484 raft_consensus.cc:3037] T ebcf1894471f4fb7b951a730c2a781dd P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:19.901664 4440 raft_consensus.cc:3037] T ebcf1894471f4fb7b951a730c2a781dd P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:19.895880 7482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcf1894471f4fb7b951a730c2a781dd" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:19.901013 8931 leader_election.cc:288] T ebcf1894471f4fb7b951a730c2a781dd P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Requested vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:19.904510 7484 raft_consensus.cc:2454] T ebcf1894471f4fb7b951a730c2a781dd P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:27:19.904568 4440 raft_consensus.cc:2454] T ebcf1894471f4fb7b951a730c2a781dd P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:27:19.905134 4508 leader_election.cc:302] T ebcf1894471f4fb7b951a730c2a781dd P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:27:19.905797 8931 raft_consensus.cc:2781] T ebcf1894471f4fb7b951a730c2a781dd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:19.905959 8931 raft_consensus.cc:686] T ebcf1894471f4fb7b951a730c2a781dd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:27:19.906221 8931 consensus_queue.cc:237] T ebcf1894471f4fb7b951a730c2a781dd P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:19.910667 4300 catalog_manager.cc:5484] T ebcf1894471f4fb7b951a730c2a781dd P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:19.938287 4300 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:19.946236 8997 tablet.cc:1722] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:19.946866 8997 tablet.cc:1620] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:19.946995 8998 tablet.cc:1722] T c29d52114111471ca87a5f41f493bbb5 P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:19.947460 8998 tablet.cc:1620] T c29d52114111471ca87a5f41f493bbb5 P 73a1f2ca1f544de9b0e3b37015f85328: MemRowSet was empty: no flush needed.
I20240502 21:27:19.950739 8999 tablet.cc:1722] T c29d52114111471ca87a5f41f493bbb5 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:19.951260 8999 tablet.cc:1620] T c29d52114111471ca87a5f41f493bbb5 P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:19.956969 4294 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=5665c045531041388f6095c33442c337] alter complete (version 2)
I20240502 21:27:19.965118 8875 raft_consensus.cc:482] T 612a0397b1754fda81c4a0019067a0d3 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:19.965354 8875 raft_consensus.cc:504] T 612a0397b1754fda81c4a0019067a0d3 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:19.966096 8875 leader_election.cc:288] T 612a0397b1754fda81c4a0019067a0d3 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:19.966593 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "612a0397b1754fda81c4a0019067a0d3" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:19.966655 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "612a0397b1754fda81c4a0019067a0d3" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:27:19.966840 4440 raft_consensus.cc:2454] T 612a0397b1754fda81c4a0019067a0d3 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:19.966842 7618 raft_consensus.cc:2454] T 612a0397b1754fda81c4a0019067a0d3 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:19.967257 7148 leader_election.cc:302] T 612a0397b1754fda81c4a0019067a0d3 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:19.967713 8875 raft_consensus.cc:2781] T 612a0397b1754fda81c4a0019067a0d3 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:19.967823 8875 raft_consensus.cc:482] T 612a0397b1754fda81c4a0019067a0d3 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:19.967937 8875 raft_consensus.cc:3037] T 612a0397b1754fda81c4a0019067a0d3 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:19.969918 8875 raft_consensus.cc:504] T 612a0397b1754fda81c4a0019067a0d3 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:19.970505 8875 leader_election.cc:288] T 612a0397b1754fda81c4a0019067a0d3 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:19.971035 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "612a0397b1754fda81c4a0019067a0d3" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:19.971278 7618 raft_consensus.cc:3037] T 612a0397b1754fda81c4a0019067a0d3 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:19.971243 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "612a0397b1754fda81c4a0019067a0d3" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:27:19.971429 4440 raft_consensus.cc:3037] T 612a0397b1754fda81c4a0019067a0d3 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:19.973667 7618 raft_consensus.cc:2454] T 612a0397b1754fda81c4a0019067a0d3 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:19.973755 4440 raft_consensus.cc:2454] T 612a0397b1754fda81c4a0019067a0d3 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:19.974092 7149 leader_election.cc:302] T 612a0397b1754fda81c4a0019067a0d3 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:19.974354 8875 raft_consensus.cc:2781] T 612a0397b1754fda81c4a0019067a0d3 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:19.974539 8875 raft_consensus.cc:686] T 612a0397b1754fda81c4a0019067a0d3 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:27:19.974820 8875 consensus_queue.cc:237] T 612a0397b1754fda81c4a0019067a0d3 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:19.978406 4300 catalog_manager.cc:5484] T 612a0397b1754fda81c4a0019067a0d3 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69). New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:20.061627 4300 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:20.068723 8997 tablet.cc:1722] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:20.069839 8999 tablet.cc:1722] T c29d52114111471ca87a5f41f493bbb5 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:20.073397 8998 tablet.cc:1722] T c29d52114111471ca87a5f41f493bbb5 P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:20.080840 4294 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=5665c045531041388f6095c33442c337] alter complete (version 3)
I20240502 21:27:20.183382 4294 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:20.183699 4294 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:20.186709 4294 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet c29d52114111471ca87a5f41f493bbb5
I20240502 21:27:20.187597 4422 tablet_service.cc:1514] Processing DeleteTablet for tablet c29d52114111471ca87a5f41f493bbb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:20 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:20.188066 4818 tablet_service.cc:1514] Processing DeleteTablet for tablet c29d52114111471ca87a5f41f493bbb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:20 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:27:20.188251 7462 tablet_service.cc:1514] Processing DeleteTablet for tablet c29d52114111471ca87a5f41f493bbb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:20 UTC) from {username='slave'} at 127.0.0.1:37182
I20240502 21:27:20.188751 9006 tablet_replica.cc:330] T c29d52114111471ca87a5f41f493bbb5 P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:27:20.189059 9006 raft_consensus.cc:2229] T c29d52114111471ca87a5f41f493bbb5 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:20.189124 9007 tablet_replica.cc:330] T c29d52114111471ca87a5f41f493bbb5 P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:27:20.189229 9005 tablet_replica.cc:330] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:20.189414 9006 raft_consensus.cc:2258] T c29d52114111471ca87a5f41f493bbb5 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:20.189496 9007 raft_consensus.cc:2229] T c29d52114111471ca87a5f41f493bbb5 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:20.189781 9007 raft_consensus.cc:2258] T c29d52114111471ca87a5f41f493bbb5 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:20.189514 9005 raft_consensus.cc:2229] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:20.190536 9005 raft_consensus.cc:2258] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:20.190865 9006 ts_tablet_manager.cc:1875] T c29d52114111471ca87a5f41f493bbb5 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:20.191197 9007 ts_tablet_manager.cc:1875] T c29d52114111471ca87a5f41f493bbb5 P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:20.191764 9005 ts_tablet_manager.cc:1875] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:20.195924 9006 ts_tablet_manager.cc:1888] T c29d52114111471ca87a5f41f493bbb5 P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:20.196043 9006 log.cc:1196] T c29d52114111471ca87a5f41f493bbb5 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/c29d52114111471ca87a5f41f493bbb5
I20240502 21:27:20.196544 9006 ts_tablet_manager.cc:1909] T c29d52114111471ca87a5f41f493bbb5 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:27:20.197407 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet c29d52114111471ca87a5f41f493bbb5 (table rebalancer_dynamic_table_new_name [id=5665c045531041388f6095c33442c337]) successfully deleted
I20240502 21:27:20.197710 9007 ts_tablet_manager.cc:1888] T c29d52114111471ca87a5f41f493bbb5 P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:20.197878 9007 log.cc:1196] T c29d52114111471ca87a5f41f493bbb5 P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/c29d52114111471ca87a5f41f493bbb5
I20240502 21:27:20.198055 9005 ts_tablet_manager.cc:1888] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:20.198212 9005 log.cc:1196] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/c29d52114111471ca87a5f41f493bbb5
I20240502 21:27:20.198431 9007 ts_tablet_manager.cc:1909] T c29d52114111471ca87a5f41f493bbb5 P 31a72286397c4edc9a63eca3412be168: Deleting consensus metadata
I20240502 21:27:20.198647 9005 ts_tablet_manager.cc:1909] T c29d52114111471ca87a5f41f493bbb5 P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:20.199410 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet c29d52114111471ca87a5f41f493bbb5 (table rebalancer_dynamic_table_new_name [id=5665c045531041388f6095c33442c337]) successfully deleted
I20240502 21:27:20.199674 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet c29d52114111471ca87a5f41f493bbb5 (table rebalancer_dynamic_table_new_name [id=5665c045531041388f6095c33442c337]) successfully deleted
I20240502 21:27:20.290304 4294 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:20.298836 4422 tablet_service.cc:1467] Processing CreateTablet for tablet d77c8fbfdae742488e4407d3fdb58dd2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6945172a1d6b4d68a027d9be835be7ea]), partition=
I20240502 21:27:20.299117 7328 tablet_service.cc:1467] Processing CreateTablet for tablet d77c8fbfdae742488e4407d3fdb58dd2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6945172a1d6b4d68a027d9be835be7ea]), partition=
I20240502 21:27:20.299293 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d77c8fbfdae742488e4407d3fdb58dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:20.299433 7462 tablet_service.cc:1467] Processing CreateTablet for tablet d77c8fbfdae742488e4407d3fdb58dd2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6945172a1d6b4d68a027d9be835be7ea]), partition=
I20240502 21:27:20.299497 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d77c8fbfdae742488e4407d3fdb58dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:20.299839 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d77c8fbfdae742488e4407d3fdb58dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:20.306135 8875 consensus_queue.cc:1035] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:20.317294 9008 tablet_bootstrap.cc:492] T d77c8fbfdae742488e4407d3fdb58dd2 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:20.322145 9008 tablet_bootstrap.cc:654] T d77c8fbfdae742488e4407d3fdb58dd2 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:20.343138 9008 tablet_bootstrap.cc:492] T d77c8fbfdae742488e4407d3fdb58dd2 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:20.343360 9008 ts_tablet_manager.cc:1367] T d77c8fbfdae742488e4407d3fdb58dd2 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.009s
I20240502 21:27:20.344177 9008 raft_consensus.cc:348] T d77c8fbfdae742488e4407d3fdb58dd2 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:20.344475 9008 raft_consensus.cc:374] T d77c8fbfdae742488e4407d3fdb58dd2 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:20.344648 9008 raft_consensus.cc:729] T d77c8fbfdae742488e4407d3fdb58dd2 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:20.345012 9008 consensus_queue.cc:260] T d77c8fbfdae742488e4407d3fdb58dd2 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:20.345777 9008 ts_tablet_manager.cc:1398] T d77c8fbfdae742488e4407d3fdb58dd2 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:20.352263 9011 tablet_bootstrap.cc:492] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:20.357035 9011 tablet_bootstrap.cc:654] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:20.360687 9009 tablet_bootstrap.cc:492] T d77c8fbfdae742488e4407d3fdb58dd2 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:20.362471 9009 tablet_bootstrap.cc:654] T d77c8fbfdae742488e4407d3fdb58dd2 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:20.362633 9011 tablet_bootstrap.cc:492] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:20.362877 9011 ts_tablet_manager.cc:1367] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.001s
I20240502 21:27:20.363782 9011 raft_consensus.cc:348] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:20.365752 9011 raft_consensus.cc:374] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:20.365885 9011 raft_consensus.cc:729] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:20.366197 9011 consensus_queue.cc:260] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:20.368320 9011 ts_tablet_manager.cc:1398] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:27:20.375128 9009 tablet_bootstrap.cc:492] T d77c8fbfdae742488e4407d3fdb58dd2 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:20.375257 8875 consensus_queue.cc:1035] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:27:20.375378 9009 ts_tablet_manager.cc:1367] T d77c8fbfdae742488e4407d3fdb58dd2 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240502 21:27:20.375416 8973 consensus_queue.cc:1035] T ebcf1894471f4fb7b951a730c2a781dd P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:20.376451 9009 raft_consensus.cc:348] T d77c8fbfdae742488e4407d3fdb58dd2 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:20.377046 9009 raft_consensus.cc:374] T d77c8fbfdae742488e4407d3fdb58dd2 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:20.377195 9009 raft_consensus.cc:729] T d77c8fbfdae742488e4407d3fdb58dd2 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:20.378000 9009 consensus_queue.cc:260] T d77c8fbfdae742488e4407d3fdb58dd2 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:20.380489 9009 ts_tablet_manager.cc:1398] T d77c8fbfdae742488e4407d3fdb58dd2 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:27:20.386194 8973 consensus_queue.cc:1035] T ebcf1894471f4fb7b951a730c2a781dd P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:20.425395 8765 raft_consensus.cc:482] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:20.425573 8765 raft_consensus.cc:504] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:20.426355 8765 leader_election.cc:288] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:20.426990 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d77c8fbfdae742488e4407d3fdb58dd2" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:20.427372 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d77c8fbfdae742488e4407d3fdb58dd2" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:20.427620 7349 raft_consensus.cc:2454] T d77c8fbfdae742488e4407d3fdb58dd2 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:20.428026 4377 leader_election.cc:302] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:20.428311 8765 raft_consensus.cc:2781] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:20.428408 8765 raft_consensus.cc:482] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:20.428485 8765 raft_consensus.cc:3037] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:20.431886 8765 raft_consensus.cc:504] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:20.432507 7484 raft_consensus.cc:2454] T d77c8fbfdae742488e4407d3fdb58dd2 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:20.432934 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d77c8fbfdae742488e4407d3fdb58dd2" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:20.433166 7349 raft_consensus.cc:3037] T d77c8fbfdae742488e4407d3fdb58dd2 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:20.433408 8765 leader_election.cc:288] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:20.435817 7349 raft_consensus.cc:2454] T d77c8fbfdae742488e4407d3fdb58dd2 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:20.433238 7482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d77c8fbfdae742488e4407d3fdb58dd2" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:20.436527 7482 raft_consensus.cc:3037] T d77c8fbfdae742488e4407d3fdb58dd2 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:20.437363 4377 leader_election.cc:302] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:20.437625 8765 raft_consensus.cc:2781] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:20.437753 8765 raft_consensus.cc:686] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:27:20.438023 8765 consensus_queue.cc:237] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:20.439184 7482 raft_consensus.cc:2454] T d77c8fbfdae742488e4407d3fdb58dd2 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:20.442363 4294 catalog_manager.cc:5484] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:27:20.497068 8875 consensus_queue.cc:1035] T 612a0397b1754fda81c4a0019067a0d3 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:20.501919 4291 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:20.502849 8875 consensus_queue.cc:1035] T 612a0397b1754fda81c4a0019067a0d3 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:20.514665 4818 tablet_service.cc:1467] Processing CreateTablet for tablet adbccf5386624b28af36ff893ac9378f (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=2e1fa2b5e59045abbc77e0d878e997ac]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:20.514652 7462 tablet_service.cc:1467] Processing CreateTablet for tablet adbccf5386624b28af36ff893ac9378f (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=2e1fa2b5e59045abbc77e0d878e997ac]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:20.515110 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adbccf5386624b28af36ff893ac9378f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:20.515146 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adbccf5386624b28af36ff893ac9378f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:20.515681 7595 tablet_service.cc:1467] Processing CreateTablet for tablet 2707427ddc3d49b89c000e92e3b663d9 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=2e1fa2b5e59045abbc77e0d878e997ac]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:20.516217 7595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2707427ddc3d49b89c000e92e3b663d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:20.520095 4555 tablet_service.cc:1467] Processing CreateTablet for tablet adbccf5386624b28af36ff893ac9378f (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=2e1fa2b5e59045abbc77e0d878e997ac]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:20.520550 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adbccf5386624b28af36ff893ac9378f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:20.522909 7194 tablet_service.cc:1467] Processing CreateTablet for tablet 2707427ddc3d49b89c000e92e3b663d9 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=2e1fa2b5e59045abbc77e0d878e997ac]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:20.523362 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2707427ddc3d49b89c000e92e3b663d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:20.525079 9027 tablet_bootstrap.cc:492] T adbccf5386624b28af36ff893ac9378f P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:20.529273 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 2707427ddc3d49b89c000e92e3b663d9 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=2e1fa2b5e59045abbc77e0d878e997ac]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:20.529781 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2707427ddc3d49b89c000e92e3b663d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:20.531752 7597 tablet_service.cc:1467] Processing CreateTablet for tablet 0c56940914cb4161bd258bf881f06c07 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=2e1fa2b5e59045abbc77e0d878e997ac]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:20.532090 9027 tablet_bootstrap.cc:654] T adbccf5386624b28af36ff893ac9378f P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:20.532218 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c56940914cb4161bd258bf881f06c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:20.536031 4818 tablet_service.cc:1467] Processing CreateTablet for tablet 0c56940914cb4161bd258bf881f06c07 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=2e1fa2b5e59045abbc77e0d878e997ac]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:20.536463 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c56940914cb4161bd258bf881f06c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:20.536559 9008 tablet_bootstrap.cc:492] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:20.538220 9008 tablet_bootstrap.cc:654] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:20.541332 9008 tablet_bootstrap.cc:492] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:20.541898 9008 ts_tablet_manager.cc:1367] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240502 21:27:20.541939 7195 tablet_service.cc:1467] Processing CreateTablet for tablet 0c56940914cb4161bd258bf881f06c07 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=2e1fa2b5e59045abbc77e0d878e997ac]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:20.542408 7195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c56940914cb4161bd258bf881f06c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:20.543190 9008 raft_consensus.cc:348] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:20.543494 9008 raft_consensus.cc:374] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:20.543816 9008 raft_consensus.cc:729] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:20.544435 9008 consensus_queue.cc:260] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:20.545357 9008 ts_tablet_manager.cc:1398] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:27:20.557776 9028 tablet_bootstrap.cc:492] T adbccf5386624b28af36ff893ac9378f P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:20.561952 9028 tablet_bootstrap.cc:654] T adbccf5386624b28af36ff893ac9378f P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:20.565541 9028 tablet_bootstrap.cc:492] T adbccf5386624b28af36ff893ac9378f P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:20.566107 9028 ts_tablet_manager.cc:1367] T adbccf5386624b28af36ff893ac9378f P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240502 21:27:20.567206 9028 raft_consensus.cc:348] T adbccf5386624b28af36ff893ac9378f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:20.570359 9027 tablet_bootstrap.cc:492] T adbccf5386624b28af36ff893ac9378f P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:20.570575 9027 ts_tablet_manager.cc:1367] T adbccf5386624b28af36ff893ac9378f P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.046s user 0.011s sys 0.017s
I20240502 21:27:20.570611 9031 tablet_bootstrap.cc:492] T 2707427ddc3d49b89c000e92e3b663d9 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:20.570866 4291 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:20.571359 9027 raft_consensus.cc:348] T adbccf5386624b28af36ff893ac9378f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:20.571533 9027 raft_consensus.cc:374] T adbccf5386624b28af36ff893ac9378f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:20.571668 9027 raft_consensus.cc:729] T adbccf5386624b28af36ff893ac9378f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:20.571903 9027 consensus_queue.cc:260] T adbccf5386624b28af36ff893ac9378f P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:20.572372 9028 raft_consensus.cc:374] T adbccf5386624b28af36ff893ac9378f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:20.572491 9028 raft_consensus.cc:729] T adbccf5386624b28af36ff893ac9378f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:20.573457 9027 ts_tablet_manager.cc:1398] T adbccf5386624b28af36ff893ac9378f P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:27:20.574069 9028 consensus_queue.cc:260] T adbccf5386624b28af36ff893ac9378f P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:20.574725 9028 ts_tablet_manager.cc:1398] T adbccf5386624b28af36ff893ac9378f P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240502 21:27:20.576172 9031 tablet_bootstrap.cc:654] T 2707427ddc3d49b89c000e92e3b663d9 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:20.585184 9031 tablet_bootstrap.cc:492] T 2707427ddc3d49b89c000e92e3b663d9 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:20.585381 9031 ts_tablet_manager.cc:1367] T 2707427ddc3d49b89c000e92e3b663d9 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20240502 21:27:20.586280 9031 raft_consensus.cc:348] T 2707427ddc3d49b89c000e92e3b663d9 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:20.586575 9031 raft_consensus.cc:374] T 2707427ddc3d49b89c000e92e3b663d9 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:20.586663 7349 raft_consensus.cc:1261] T d77c8fbfdae742488e4407d3fdb58dd2 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:20.586717 9031 raft_consensus.cc:729] T 2707427ddc3d49b89c000e92e3b663d9 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:20.586879 7482 raft_consensus.cc:1261] T d77c8fbfdae742488e4407d3fdb58dd2 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:20.587087 9031 consensus_queue.cc:260] T 2707427ddc3d49b89c000e92e3b663d9 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:20.587774 9036 consensus_queue.cc:1035] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:20.589108 9031 ts_tablet_manager.cc:1398] T 2707427ddc3d49b89c000e92e3b663d9 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:20.590966 9036 consensus_queue.cc:1035] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:20.598407 4291 catalog_manager.cc:5576] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719) does not have the latest schema for tablet d77c8fbfdae742488e4407d3fdb58dd2 (table rebalancer_dynamic_table [id=6945172a1d6b4d68a027d9be835be7ea]). Expected version 1 got 0
I20240502 21:27:20.601544 9041 tablet.cc:1722] T d77c8fbfdae742488e4407d3fdb58dd2 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:20.602567 9041 tablet.cc:1620] T d77c8fbfdae742488e4407d3fdb58dd2 P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:20.603740 4291 catalog_manager.cc:5576] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719) does not have the latest schema for tablet d77c8fbfdae742488e4407d3fdb58dd2 (table rebalancer_dynamic_table [id=6945172a1d6b4d68a027d9be835be7ea]). Expected version 1 got 0
I20240502 21:27:20.607926 9040 tablet.cc:1722] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:20.609651 9040 tablet.cc:1620] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:20.612547 9027 tablet_bootstrap.cc:492] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:20.614209 9027 tablet_bootstrap.cc:654] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:20.616668 9027 tablet_bootstrap.cc:492] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:20.617009 9027 ts_tablet_manager.cc:1367] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 21:27:20.623106 9031 tablet_bootstrap.cc:492] T 0c56940914cb4161bd258bf881f06c07 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:20.624908 9027 raft_consensus.cc:348] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:20.625243 9027 raft_consensus.cc:374] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:20.625380 9027 raft_consensus.cc:729] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:20.625789 9031 tablet_bootstrap.cc:654] T 0c56940914cb4161bd258bf881f06c07 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:20.625730 9027 consensus_queue.cc:260] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:20.626488 9027 ts_tablet_manager.cc:1398] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240502 21:27:20.626794 4294 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=6945172a1d6b4d68a027d9be835be7ea] alter complete (version 1)
I20240502 21:27:20.631775 8852 raft_consensus.cc:482] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:20.631945 8852 raft_consensus.cc:504] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:20.632822 8852 leader_election.cc:288] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:27:20.633280 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adbccf5386624b28af36ff893ac9378f" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:27:20.633507 4839 raft_consensus.cc:2454] T adbccf5386624b28af36ff893ac9378f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:20.633996 7416 leader_election.cc:302] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:20.634704 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adbccf5386624b28af36ff893ac9378f" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:27:20.634974 4574 raft_consensus.cc:2454] T adbccf5386624b28af36ff893ac9378f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:20.635995 8852 raft_consensus.cc:2781] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:20.636122 8852 raft_consensus.cc:482] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:20.636220 8852 raft_consensus.cc:3037] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:20.642033 8852 raft_consensus.cc:504] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:20.642316 9033 tablet_bootstrap.cc:492] T 2707427ddc3d49b89c000e92e3b663d9 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:20.642822 8852 leader_election.cc:288] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Requested vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:27:20.643113 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adbccf5386624b28af36ff893ac9378f" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:27:20.643126 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adbccf5386624b28af36ff893ac9378f" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:27:20.643303 4574 raft_consensus.cc:3037] T adbccf5386624b28af36ff893ac9378f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:20.643303 4839 raft_consensus.cc:3037] T adbccf5386624b28af36ff893ac9378f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:20.644136 9030 tablet_bootstrap.cc:492] T 2707427ddc3d49b89c000e92e3b663d9 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:20.645673 9031 tablet_bootstrap.cc:492] T 0c56940914cb4161bd258bf881f06c07 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:20.645846 9031 ts_tablet_manager.cc:1367] T 0c56940914cb4161bd258bf881f06c07 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.003s
I20240502 21:27:20.646008 9030 tablet_bootstrap.cc:654] T 2707427ddc3d49b89c000e92e3b663d9 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:20.646103 4839 raft_consensus.cc:2454] T adbccf5386624b28af36ff893ac9378f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:20.646581 4574 raft_consensus.cc:2454] T adbccf5386624b28af36ff893ac9378f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:20.646776 9031 raft_consensus.cc:348] T 0c56940914cb4161bd258bf881f06c07 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:20.647217 9031 raft_consensus.cc:374] T 0c56940914cb4161bd258bf881f06c07 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:20.647243 7416 leader_election.cc:302] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:20.647614 9037 raft_consensus.cc:2781] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:20.647895 9037 raft_consensus.cc:686] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Becoming Leader. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Running, Role: LEADER
I20240502 21:27:20.647460 9031 raft_consensus.cc:729] T 0c56940914cb4161bd258bf881f06c07 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:20.648267 9037 consensus_queue.cc:237] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:20.649212 9031 consensus_queue.cc:260] T 0c56940914cb4161bd258bf881f06c07 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:20.649391 9033 tablet_bootstrap.cc:654] T 2707427ddc3d49b89c000e92e3b663d9 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:20.650110 9031 ts_tablet_manager.cc:1398] T 0c56940914cb4161bd258bf881f06c07 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 21:27:20.652778 9030 tablet_bootstrap.cc:492] T 2707427ddc3d49b89c000e92e3b663d9 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:27:20.653019 9033 tablet_bootstrap.cc:492] T 2707427ddc3d49b89c000e92e3b663d9 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:20.653211 9033 ts_tablet_manager.cc:1367] T 2707427ddc3d49b89c000e92e3b663d9 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240502 21:27:20.654274 9033 raft_consensus.cc:348] T 2707427ddc3d49b89c000e92e3b663d9 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:20.654594 9033 raft_consensus.cc:374] T 2707427ddc3d49b89c000e92e3b663d9 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:20.654659 9030 ts_tablet_manager.cc:1367] T 2707427ddc3d49b89c000e92e3b663d9 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240502 21:27:20.654731 9033 raft_consensus.cc:729] T 2707427ddc3d49b89c000e92e3b663d9 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:20.655066 9033 consensus_queue.cc:260] T 2707427ddc3d49b89c000e92e3b663d9 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:20.655823 9033 ts_tablet_manager.cc:1398] T 2707427ddc3d49b89c000e92e3b663d9 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:20.656139 9030 raft_consensus.cc:348] T 2707427ddc3d49b89c000e92e3b663d9 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:20.656421 9030 raft_consensus.cc:374] T 2707427ddc3d49b89c000e92e3b663d9 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:20.656579 9030 raft_consensus.cc:729] T 2707427ddc3d49b89c000e92e3b663d9 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:20.656725 9033 tablet_bootstrap.cc:492] T 0c56940914cb4161bd258bf881f06c07 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:20.657011 9030 consensus_queue.cc:260] T 2707427ddc3d49b89c000e92e3b663d9 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:20.657768 4292 catalog_manager.cc:5576] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913) does not have the latest schema for tablet d77c8fbfdae742488e4407d3fdb58dd2 (table rebalancer_dynamic_table [id=6945172a1d6b4d68a027d9be835be7ea]). Expected version 1 got 0
I20240502 21:27:20.658378 9033 tablet_bootstrap.cc:654] T 0c56940914cb4161bd258bf881f06c07 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:20.659168 9030 ts_tablet_manager.cc:1398] T 2707427ddc3d49b89c000e92e3b663d9 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:20.662616 9033 tablet_bootstrap.cc:492] T 0c56940914cb4161bd258bf881f06c07 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:20.662825 9033 ts_tablet_manager.cc:1367] T 0c56940914cb4161bd258bf881f06c07 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 21:27:20.663666 9033 raft_consensus.cc:348] T 0c56940914cb4161bd258bf881f06c07 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:20.663960 9033 raft_consensus.cc:374] T 0c56940914cb4161bd258bf881f06c07 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:20.664098 9033 raft_consensus.cc:729] T 0c56940914cb4161bd258bf881f06c07 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:20.664415 9033 consensus_queue.cc:260] T 0c56940914cb4161bd258bf881f06c07 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:20.665108 9033 ts_tablet_manager.cc:1398] T 0c56940914cb4161bd258bf881f06c07 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:20.667948 9048 tablet.cc:1722] T d77c8fbfdae742488e4407d3fdb58dd2 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:20.668812 9048 tablet.cc:1620] T d77c8fbfdae742488e4407d3fdb58dd2 P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:20.670017 4294 catalog_manager.cc:5576] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913) does not have the latest schema for tablet d77c8fbfdae742488e4407d3fdb58dd2 (table rebalancer_dynamic_table [id=6945172a1d6b4d68a027d9be835be7ea]). Expected version 1 got 0
I20240502 21:27:20.670300 4294 catalog_manager.cc:5484] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a72286397c4edc9a63eca3412be168 (127.29.82.71). New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:27:20.708407 8834 raft_consensus.cc:482] T 2707427ddc3d49b89c000e92e3b663d9 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:20.708626 8834 raft_consensus.cc:504] T 2707427ddc3d49b89c000e92e3b663d9 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:20.709400 8834 leader_election.cc:288] T 2707427ddc3d49b89c000e92e3b663d9 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:27:20.709929 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2707427ddc3d49b89c000e92e3b663d9" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:27:20.710188 4706 raft_consensus.cc:2454] T 2707427ddc3d49b89c000e92e3b663d9 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:20.710237 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2707427ddc3d49b89c000e92e3b663d9" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:20.710479 7215 raft_consensus.cc:2454] T 2707427ddc3d49b89c000e92e3b663d9 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:20.710851 7552 leader_election.cc:302] T 2707427ddc3d49b89c000e92e3b663d9 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 398c3c8f3195492c900439ad5e1931bb; no voters:
I20240502 21:27:20.711287 8834 raft_consensus.cc:2781] T 2707427ddc3d49b89c000e92e3b663d9 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:20.711421 8834 raft_consensus.cc:482] T 2707427ddc3d49b89c000e92e3b663d9 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:20.711524 8834 raft_consensus.cc:3037] T 2707427ddc3d49b89c000e92e3b663d9 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:20.713721 8834 raft_consensus.cc:504] T 2707427ddc3d49b89c000e92e3b663d9 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:20.714267 8834 leader_election.cc:288] T 2707427ddc3d49b89c000e92e3b663d9 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Requested vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:27:20.714707 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2707427ddc3d49b89c000e92e3b663d9" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:20.714763 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2707427ddc3d49b89c000e92e3b663d9" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:27:20.714957 4706 raft_consensus.cc:3037] T 2707427ddc3d49b89c000e92e3b663d9 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:20.714964 7215 raft_consensus.cc:3037] T 2707427ddc3d49b89c000e92e3b663d9 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:20.717217 4706 raft_consensus.cc:2454] T 2707427ddc3d49b89c000e92e3b663d9 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:20.717294 7215 raft_consensus.cc:2454] T 2707427ddc3d49b89c000e92e3b663d9 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:20.718099 7552 leader_election.cc:302] T 2707427ddc3d49b89c000e92e3b663d9 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 398c3c8f3195492c900439ad5e1931bb, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:20.718439 8834 raft_consensus.cc:2781] T 2707427ddc3d49b89c000e92e3b663d9 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:20.718585 8834 raft_consensus.cc:686] T 2707427ddc3d49b89c000e92e3b663d9 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Becoming Leader. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Running, Role: LEADER
I20240502 21:27:20.718868 8834 consensus_queue.cc:237] T 2707427ddc3d49b89c000e92e3b663d9 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:20.722513 4294 catalog_manager.cc:5484] T 2707427ddc3d49b89c000e92e3b663d9 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ddd486109f5434cace13d60d633ac04 (127.29.82.72). New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:20.737354 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:20.746735 9040 tablet.cc:1722] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:20.747237 9040 tablet.cc:1620] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:20.752238 9041 tablet.cc:1722] T d77c8fbfdae742488e4407d3fdb58dd2 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:20.752730 9041 tablet.cc:1620] T d77c8fbfdae742488e4407d3fdb58dd2 P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:20.761920 4294 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=6945172a1d6b4d68a027d9be835be7ea] alter complete (version 2)
I20240502 21:27:20.766731 9048 tablet.cc:1722] T d77c8fbfdae742488e4407d3fdb58dd2 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:20.767266 9048 tablet.cc:1620] T d77c8fbfdae742488e4407d3fdb58dd2 P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:20.866452 4300 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:20.878907 9040 tablet.cc:1722] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:20.879699 9041 tablet.cc:1722] T d77c8fbfdae742488e4407d3fdb58dd2 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:20.888065 4300 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=6945172a1d6b4d68a027d9be835be7ea] alter complete (version 3)
I20240502 21:27:20.892490 9048 tablet.cc:1722] T d77c8fbfdae742488e4407d3fdb58dd2 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:20.909977 8561 raft_consensus.cc:482] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:20.910143 8561 raft_consensus.cc:504] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:20.910964 8561 leader_election.cc:288] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:20.911249 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c56940914cb4161bd258bf881f06c07" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:20.911466 7215 raft_consensus.cc:2454] T 0c56940914cb4161bd258bf881f06c07 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:27:20.911459 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c56940914cb4161bd258bf881f06c07" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:20.911774 7618 raft_consensus.cc:2454] T 0c56940914cb4161bd258bf881f06c07 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:27:20.912007 4772 leader_election.cc:302] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:20.912510 8889 raft_consensus.cc:2781] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:20.912609 8889 raft_consensus.cc:482] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:20.912717 8889 raft_consensus.cc:3037] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:20.915294 8889 raft_consensus.cc:504] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:20.915885 8889 leader_election.cc:288] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:20.916273 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c56940914cb4161bd258bf881f06c07" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:20.916282 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c56940914cb4161bd258bf881f06c07" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:20.916437 7215 raft_consensus.cc:3037] T 0c56940914cb4161bd258bf881f06c07 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:20.916450 7618 raft_consensus.cc:3037] T 0c56940914cb4161bd258bf881f06c07 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:20.918365 7215 raft_consensus.cc:2454] T 0c56940914cb4161bd258bf881f06c07 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:27:20.918555 7618 raft_consensus.cc:2454] T 0c56940914cb4161bd258bf881f06c07 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:27:20.918754 4772 leader_election.cc:302] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:20.919055 8889 raft_consensus.cc:2781] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:20.919265 8889 raft_consensus.cc:686] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:27:20.919595 8889 consensus_queue.cc:237] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:20.923130 4300 catalog_manager.cc:5484] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:20.991981 4300 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:20.992290 4300 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:20.995765 4300 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet d77c8fbfdae742488e4407d3fdb58dd2
I20240502 21:27:20.996577 7328 tablet_service.cc:1514] Processing DeleteTablet for tablet d77c8fbfdae742488e4407d3fdb58dd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:20 UTC) from {username='slave'} at 127.0.0.1:47078
I20240502 21:27:20.997114 7462 tablet_service.cc:1514] Processing DeleteTablet for tablet d77c8fbfdae742488e4407d3fdb58dd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:20 UTC) from {username='slave'} at 127.0.0.1:37182
I20240502 21:27:20.997545 9054 tablet_replica.cc:330] T d77c8fbfdae742488e4407d3fdb58dd2 P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:27:20.997571 4422 tablet_service.cc:1514] Processing DeleteTablet for tablet d77c8fbfdae742488e4407d3fdb58dd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:20 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:20.997730 9055 tablet_replica.cc:330] T d77c8fbfdae742488e4407d3fdb58dd2 P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:27:20.997968 9055 raft_consensus.cc:2229] T d77c8fbfdae742488e4407d3fdb58dd2 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:20.998225 9056 tablet_replica.cc:330] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:20.998263 9055 raft_consensus.cc:2258] T d77c8fbfdae742488e4407d3fdb58dd2 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:20.998855 9056 raft_consensus.cc:2229] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:20.999230 9055 ts_tablet_manager.cc:1875] T d77c8fbfdae742488e4407d3fdb58dd2 P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:20.999280 9056 raft_consensus.cc:2258] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:21.002480 9054 raft_consensus.cc:2229] T d77c8fbfdae742488e4407d3fdb58dd2 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:21.002779 9054 raft_consensus.cc:2258] T d77c8fbfdae742488e4407d3fdb58dd2 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:21.004238 9054 ts_tablet_manager.cc:1875] T d77c8fbfdae742488e4407d3fdb58dd2 P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:21.004595 9056 ts_tablet_manager.cc:1875] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:21.004853 9055 ts_tablet_manager.cc:1888] T d77c8fbfdae742488e4407d3fdb58dd2 P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:21.005033 9055 log.cc:1196] T d77c8fbfdae742488e4407d3fdb58dd2 P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/d77c8fbfdae742488e4407d3fdb58dd2
I20240502 21:27:21.005704 9055 ts_tablet_manager.cc:1909] T d77c8fbfdae742488e4407d3fdb58dd2 P 31a72286397c4edc9a63eca3412be168: Deleting consensus metadata
I20240502 21:27:21.006762 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet d77c8fbfdae742488e4407d3fdb58dd2 (table rebalancer_dynamic_table_new_name [id=6945172a1d6b4d68a027d9be835be7ea]) successfully deleted
I20240502 21:27:21.015400 9056 ts_tablet_manager.cc:1888] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:21.015559 9056 log.cc:1196] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/d77c8fbfdae742488e4407d3fdb58dd2
I20240502 21:27:21.016044 9056 ts_tablet_manager.cc:1909] T d77c8fbfdae742488e4407d3fdb58dd2 P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:21.017328 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet d77c8fbfdae742488e4407d3fdb58dd2 (table rebalancer_dynamic_table_new_name [id=6945172a1d6b4d68a027d9be835be7ea]) successfully deleted
I20240502 21:27:21.020001 9054 ts_tablet_manager.cc:1888] T d77c8fbfdae742488e4407d3fdb58dd2 P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:21.020143 9054 log.cc:1196] T d77c8fbfdae742488e4407d3fdb58dd2 P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/d77c8fbfdae742488e4407d3fdb58dd2
I20240502 21:27:21.020591 9054 ts_tablet_manager.cc:1909] T d77c8fbfdae742488e4407d3fdb58dd2 P d0456c5e55a542128688904e003d403c: Deleting consensus metadata
I20240502 21:27:21.021721 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet d77c8fbfdae742488e4407d3fdb58dd2 (table rebalancer_dynamic_table_new_name [id=6945172a1d6b4d68a027d9be835be7ea]) successfully deleted
I20240502 21:27:21.101823 4300 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:21.111500 4555 tablet_service.cc:1467] Processing CreateTablet for tablet f24a85c6fde64fdaa228561a48cb6dc4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=176c8f5bcca5418fa720abe7cf080321]), partition=
I20240502 21:27:21.111487 7328 tablet_service.cc:1467] Processing CreateTablet for tablet f24a85c6fde64fdaa228561a48cb6dc4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=176c8f5bcca5418fa720abe7cf080321]), partition=
I20240502 21:27:21.111485 4686 tablet_service.cc:1467] Processing CreateTablet for tablet f24a85c6fde64fdaa228561a48cb6dc4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=176c8f5bcca5418fa720abe7cf080321]), partition=
I20240502 21:27:21.111948 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f24a85c6fde64fdaa228561a48cb6dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:21.111999 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f24a85c6fde64fdaa228561a48cb6dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:21.112012 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f24a85c6fde64fdaa228561a48cb6dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:21.117995 9037 consensus_queue.cc:1035] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:21.120465 9030 tablet_bootstrap.cc:492] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:21.122169 9030 tablet_bootstrap.cc:654] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:21.122469 9058 tablet_bootstrap.cc:492] T f24a85c6fde64fdaa228561a48cb6dc4 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:21.123960 9058 tablet_bootstrap.cc:654] T f24a85c6fde64fdaa228561a48cb6dc4 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:21.127889 9058 tablet_bootstrap.cc:492] T f24a85c6fde64fdaa228561a48cb6dc4 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:21.127938 8852 consensus_queue.cc:1035] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:21.128101 9058 ts_tablet_manager.cc:1367] T f24a85c6fde64fdaa228561a48cb6dc4 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 21:27:21.129045 9058 raft_consensus.cc:348] T f24a85c6fde64fdaa228561a48cb6dc4 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:21.129375 9058 raft_consensus.cc:374] T f24a85c6fde64fdaa228561a48cb6dc4 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:21.129706 9058 raft_consensus.cc:729] T f24a85c6fde64fdaa228561a48cb6dc4 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:21.130151 9058 consensus_queue.cc:260] T f24a85c6fde64fdaa228561a48cb6dc4 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:21.131726 9058 ts_tablet_manager.cc:1398] T f24a85c6fde64fdaa228561a48cb6dc4 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:21.141278 9030 tablet_bootstrap.cc:492] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:27:21.141687 9030 ts_tablet_manager.cc:1367] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240502 21:27:21.142660 9030 raft_consensus.cc:348] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:21.144335 9030 raft_consensus.cc:374] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:21.144877 9030 raft_consensus.cc:729] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:21.145264 9063 tablet_bootstrap.cc:492] T f24a85c6fde64fdaa228561a48cb6dc4 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:21.145144 9030 consensus_queue.cc:260] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:21.145898 9030 ts_tablet_manager.cc:1398] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:21.147666 9063 tablet_bootstrap.cc:654] T f24a85c6fde64fdaa228561a48cb6dc4 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:21.161219 9052 raft_consensus.cc:482] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:21.161422 9052 raft_consensus.cc:504] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:21.162575 9052 leader_election.cc:288] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:27:21.164111 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f24a85c6fde64fdaa228561a48cb6dc4" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
W20240502 21:27:21.164695 4642 leader_election.cc:341] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:21.168424 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f24a85c6fde64fdaa228561a48cb6dc4" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:21.168735 7350 raft_consensus.cc:2454] T f24a85c6fde64fdaa228561a48cb6dc4 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:27:21.169447 4641 leader_election.cc:302] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, d0456c5e55a542128688904e003d403c; no voters: 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:27:21.170037 9052 raft_consensus.cc:2781] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:21.170051 8834 consensus_queue.cc:1035] T 2707427ddc3d49b89c000e92e3b663d9 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:21.170188 9052 raft_consensus.cc:482] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:21.170293 9052 raft_consensus.cc:3037] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:21.198545 9052 raft_consensus.cc:504] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:21.199785 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f24a85c6fde64fdaa228561a48cb6dc4" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:21.200333 7350 raft_consensus.cc:3037] T f24a85c6fde64fdaa228561a48cb6dc4 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:21.201978 9052 leader_election.cc:288] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:27:21.209384 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f24a85c6fde64fdaa228561a48cb6dc4" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
W20240502 21:27:21.210036 4642 leader_election.cc:341] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:21.225415 7350 raft_consensus.cc:2454] T f24a85c6fde64fdaa228561a48cb6dc4 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:27:21.226121 4641 leader_election.cc:302] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, d0456c5e55a542128688904e003d403c; no voters: 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:27:21.226840 9052 raft_consensus.cc:2781] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:21.227208 9052 raft_consensus.cc:686] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Becoming Leader. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Running, Role: LEADER
I20240502 21:27:21.227553 9052 consensus_queue.cc:237] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:21.242208 9063 tablet_bootstrap.cc:492] T f24a85c6fde64fdaa228561a48cb6dc4 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:21.242537 9063 ts_tablet_manager.cc:1367] T f24a85c6fde64fdaa228561a48cb6dc4 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.097s user 0.006s sys 0.003s
I20240502 21:27:21.243857 9063 raft_consensus.cc:348] T f24a85c6fde64fdaa228561a48cb6dc4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:21.244222 9063 raft_consensus.cc:374] T f24a85c6fde64fdaa228561a48cb6dc4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:21.244385 9063 raft_consensus.cc:729] T f24a85c6fde64fdaa228561a48cb6dc4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:21.257802 4294 catalog_manager.cc:5484] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 398c3c8f3195492c900439ad5e1931bb (127.29.82.67). New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:21.259409 9063 consensus_queue.cc:260] T f24a85c6fde64fdaa228561a48cb6dc4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:21.262336 9063 ts_tablet_manager.cc:1398] T f24a85c6fde64fdaa228561a48cb6dc4 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
W20240502 21:27:21.263144 7551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.69:33193, user_credentials={real_user=slave}} blocked reactor thread for 79238.4us
I20240502 21:27:21.265800 8834 consensus_queue.cc:1035] T 2707427ddc3d49b89c000e92e3b663d9 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:21.301918 8889 consensus_queue.cc:1035] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:21.308079 8889 consensus_queue.cc:1035] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:21.364966 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:21.376133 4575 raft_consensus.cc:3037] T f24a85c6fde64fdaa228561a48cb6dc4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:21.379412 7350 raft_consensus.cc:1261] T f24a85c6fde64fdaa228561a48cb6dc4 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:21.389436 9052 consensus_queue.cc:1035] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240502 21:27:21.398725 4575 raft_consensus.cc:1261] T f24a85c6fde64fdaa228561a48cb6dc4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:21.399708 9052 consensus_queue.cc:1035] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:21.422876 4294 catalog_manager.cc:5576] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781) does not have the latest schema for tablet f24a85c6fde64fdaa228561a48cb6dc4 (table rebalancer_dynamic_table [id=176c8f5bcca5418fa720abe7cf080321]). Expected version 1 got 0
I20240502 21:27:21.431993 4291 catalog_manager.cc:5576] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719) does not have the latest schema for tablet f24a85c6fde64fdaa228561a48cb6dc4 (table rebalancer_dynamic_table [id=176c8f5bcca5418fa720abe7cf080321]). Expected version 1 got 0
I20240502 21:27:21.432320 9086 tablet.cc:1722] T f24a85c6fde64fdaa228561a48cb6dc4 P 7a736a8ebb164ce0a051c0fccd76c3b8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:21.433163 9086 tablet.cc:1620] T f24a85c6fde64fdaa228561a48cb6dc4 P 7a736a8ebb164ce0a051c0fccd76c3b8: MemRowSet was empty: no flush needed.
I20240502 21:27:21.434033 9087 tablet.cc:1722] T f24a85c6fde64fdaa228561a48cb6dc4 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:21.434603 9087 tablet.cc:1620] T f24a85c6fde64fdaa228561a48cb6dc4 P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:21.436149 4291 catalog_manager.cc:5576] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781) does not have the latest schema for tablet f24a85c6fde64fdaa228561a48cb6dc4 (table rebalancer_dynamic_table [id=176c8f5bcca5418fa720abe7cf080321]). Expected version 1 got 0
I20240502 21:27:21.445194 9085 tablet.cc:1722] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:21.447652 9085 tablet.cc:1620] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb: MemRowSet was empty: no flush needed.
I20240502 21:27:21.458703 4294 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=176c8f5bcca5418fa720abe7cf080321] alter complete (version 1)
I20240502 21:27:21.462229 4291 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:21.475555 7328 tablet_service.cc:1467] Processing CreateTablet for tablet 94e66998ede944a6b00478270029ebcf (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=f33bc94e9d8b4f11aa285b5624424d4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:21.476295 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e66998ede944a6b00478270029ebcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:21.476609 4422 tablet_service.cc:1467] Processing CreateTablet for tablet 94e66998ede944a6b00478270029ebcf (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=f33bc94e9d8b4f11aa285b5624424d4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:21.477022 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e66998ede944a6b00478270029ebcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:21.480352 4818 tablet_service.cc:1467] Processing CreateTablet for tablet 93e0911f163f4629a4df0249c5eb8860 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=f33bc94e9d8b4f11aa285b5624424d4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:21.480916 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93e0911f163f4629a4df0249c5eb8860. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:21.483215 7194 tablet_service.cc:1467] Processing CreateTablet for tablet 94e66998ede944a6b00478270029ebcf (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=f33bc94e9d8b4f11aa285b5624424d4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:21.483700 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e66998ede944a6b00478270029ebcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:21.484717 9058 tablet_bootstrap.cc:492] T 94e66998ede944a6b00478270029ebcf P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:21.486233 9058 tablet_bootstrap.cc:654] T 94e66998ede944a6b00478270029ebcf P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:21.488379 7462 tablet_service.cc:1467] Processing CreateTablet for tablet 93e0911f163f4629a4df0249c5eb8860 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=f33bc94e9d8b4f11aa285b5624424d4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:21.488804 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93e0911f163f4629a4df0249c5eb8860. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:21.498215 7195 tablet_service.cc:1467] Processing CreateTablet for tablet 93e0911f163f4629a4df0249c5eb8860 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=f33bc94e9d8b4f11aa285b5624424d4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:21.498411 4421 tablet_service.cc:1467] Processing CreateTablet for tablet 20b714b400824ecf86890f92aeff19be (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=f33bc94e9d8b4f11aa285b5624424d4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:21.498663 7195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93e0911f163f4629a4df0249c5eb8860. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:21.498795 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20b714b400824ecf86890f92aeff19be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:21.498970 9058 tablet_bootstrap.cc:492] T 94e66998ede944a6b00478270029ebcf P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:21.499158 9058 ts_tablet_manager.cc:1367] T 94e66998ede944a6b00478270029ebcf P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240502 21:27:21.499987 9058 raft_consensus.cc:348] T 94e66998ede944a6b00478270029ebcf P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:21.500250 9058 raft_consensus.cc:374] T 94e66998ede944a6b00478270029ebcf P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:21.500329 9058 raft_consensus.cc:729] T 94e66998ede944a6b00478270029ebcf P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:21.500602 9058 consensus_queue.cc:260] T 94e66998ede944a6b00478270029ebcf P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:21.501173 9058 ts_tablet_manager.cc:1398] T 94e66998ede944a6b00478270029ebcf P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:21.503743 9090 tablet_bootstrap.cc:492] T 94e66998ede944a6b00478270029ebcf P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:21.504372 9089 tablet_bootstrap.cc:492] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:21.507043 9089 tablet_bootstrap.cc:654] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:21.507619 7463 tablet_service.cc:1467] Processing CreateTablet for tablet 20b714b400824ecf86890f92aeff19be (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=f33bc94e9d8b4f11aa285b5624424d4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:21.508152 7463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20b714b400824ecf86890f92aeff19be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:21.510015 9090 tablet_bootstrap.cc:654] T 94e66998ede944a6b00478270029ebcf P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:21.512501 4817 tablet_service.cc:1467] Processing CreateTablet for tablet 20b714b400824ecf86890f92aeff19be (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=f33bc94e9d8b4f11aa285b5624424d4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:21.512974 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20b714b400824ecf86890f92aeff19be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:21.521154 9091 tablet_bootstrap.cc:492] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:21.523854 9091 tablet_bootstrap.cc:654] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:21.528833 9089 tablet_bootstrap.cc:492] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:21.529654 9091 tablet_bootstrap.cc:492] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:21.529731 9089 ts_tablet_manager.cc:1367] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.002s
I20240502 21:27:21.529917 9091 ts_tablet_manager.cc:1367] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240502 21:27:21.530658 9091 raft_consensus.cc:348] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:21.530874 9091 raft_consensus.cc:374] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:21.530963 9091 raft_consensus.cc:729] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:21.531211 9091 consensus_queue.cc:260] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:21.532224 9092 tablet_bootstrap.cc:492] T 93e0911f163f4629a4df0249c5eb8860 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:21.532145 9089 raft_consensus.cc:348] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:21.532438 9089 raft_consensus.cc:374] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:21.532586 9089 raft_consensus.cc:729] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:21.532755 9091 ts_tablet_manager.cc:1398] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:21.533067 9091 tablet_bootstrap.cc:492] T 20b714b400824ecf86890f92aeff19be P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:21.533248 9089 consensus_queue.cc:260] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:21.533946 9089 ts_tablet_manager.cc:1398] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 21:27:21.534787 9091 tablet_bootstrap.cc:654] T 20b714b400824ecf86890f92aeff19be P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:21.535018 9090 tablet_bootstrap.cc:492] T 94e66998ede944a6b00478270029ebcf P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:21.535265 9090 ts_tablet_manager.cc:1367] T 94e66998ede944a6b00478270029ebcf P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.000s
I20240502 21:27:21.536119 9090 raft_consensus.cc:348] T 94e66998ede944a6b00478270029ebcf P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:21.536501 9090 raft_consensus.cc:374] T 94e66998ede944a6b00478270029ebcf P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:21.537088 9092 tablet_bootstrap.cc:654] T 93e0911f163f4629a4df0249c5eb8860 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:21.539166 9090 raft_consensus.cc:729] T 94e66998ede944a6b00478270029ebcf P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:21.539791 9089 tablet_bootstrap.cc:492] T 93e0911f163f4629a4df0249c5eb8860 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:21.540115 9090 consensus_queue.cc:260] T 94e66998ede944a6b00478270029ebcf P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:21.542950 9090 ts_tablet_manager.cc:1398] T 94e66998ede944a6b00478270029ebcf P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 21:27:21.543962 9089 tablet_bootstrap.cc:654] T 93e0911f163f4629a4df0249c5eb8860 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:21.544271 9090 tablet_bootstrap.cc:492] T 20b714b400824ecf86890f92aeff19be P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:21.546029 9090 tablet_bootstrap.cc:654] T 20b714b400824ecf86890f92aeff19be P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:21.556008 8561 raft_consensus.cc:482] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:21.556254 9092 tablet_bootstrap.cc:492] T 93e0911f163f4629a4df0249c5eb8860 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:21.556210 8561 raft_consensus.cc:504] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:21.556524 9092 ts_tablet_manager.cc:1367] T 93e0911f163f4629a4df0249c5eb8860 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.010s
I20240502 21:27:21.557161 8561 leader_election.cc:288] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:21.557644 9089 tablet_bootstrap.cc:492] T 93e0911f163f4629a4df0249c5eb8860 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:21.557804 9089 ts_tablet_manager.cc:1367] T 93e0911f163f4629a4df0249c5eb8860 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.002s
I20240502 21:27:21.557859 7482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e0911f163f4629a4df0249c5eb8860" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:21.558281 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e0911f163f4629a4df0249c5eb8860" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:21.558630 9089 raft_consensus.cc:348] T 93e0911f163f4629a4df0249c5eb8860 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:21.558851 9089 raft_consensus.cc:374] T 93e0911f163f4629a4df0249c5eb8860 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:21.558941 9089 raft_consensus.cc:729] T 93e0911f163f4629a4df0249c5eb8860 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
W20240502 21:27:21.560056 4772 leader_election.cc:341] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:21.560544 9091 tablet_bootstrap.cc:492] T 20b714b400824ecf86890f92aeff19be P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:21.560756 9091 ts_tablet_manager.cc:1367] T 20b714b400824ecf86890f92aeff19be P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.020s
I20240502 21:27:21.560817 9089 consensus_queue.cc:260] T 93e0911f163f4629a4df0249c5eb8860 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:21.561434 9089 ts_tablet_manager.cc:1398] T 93e0911f163f4629a4df0249c5eb8860 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:21.561952 9091 raft_consensus.cc:348] T 20b714b400824ecf86890f92aeff19be P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:21.562206 9090 tablet_bootstrap.cc:492] T 20b714b400824ecf86890f92aeff19be P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:21.562203 9091 raft_consensus.cc:374] T 20b714b400824ecf86890f92aeff19be P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:21.562297 9091 raft_consensus.cc:729] T 20b714b400824ecf86890f92aeff19be P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:21.562393 9090 ts_tablet_manager.cc:1367] T 20b714b400824ecf86890f92aeff19be P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20240502 21:27:21.562618 9091 consensus_queue.cc:260] T 20b714b400824ecf86890f92aeff19be P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:21.563086 9090 raft_consensus.cc:348] T 20b714b400824ecf86890f92aeff19be P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:21.563292 9090 raft_consensus.cc:374] T 20b714b400824ecf86890f92aeff19be P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:21.563349 9090 raft_consensus.cc:729] T 20b714b400824ecf86890f92aeff19be P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:21.563539 9091 ts_tablet_manager.cc:1398] T 20b714b400824ecf86890f92aeff19be P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:27:21.563547 9090 consensus_queue.cc:260] T 20b714b400824ecf86890f92aeff19be P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:21.564275 9090 ts_tablet_manager.cc:1398] T 20b714b400824ecf86890f92aeff19be P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:27:21.565470 9092 raft_consensus.cc:348] T 93e0911f163f4629a4df0249c5eb8860 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:21.565815 9092 raft_consensus.cc:374] T 93e0911f163f4629a4df0249c5eb8860 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:21.565924 9092 raft_consensus.cc:729] T 93e0911f163f4629a4df0249c5eb8860 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:21.566176 9092 consensus_queue.cc:260] T 93e0911f163f4629a4df0249c5eb8860 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:21.566525 7482 raft_consensus.cc:2454] T 93e0911f163f4629a4df0249c5eb8860 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:27:21.566824 9092 ts_tablet_manager.cc:1398] T 93e0911f163f4629a4df0249c5eb8860 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.010s user 0.000s sys 0.001s
I20240502 21:27:21.567147 9092 tablet_bootstrap.cc:492] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:21.568478 4770 leader_election.cc:302] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 73a1f2ca1f544de9b0e3b37015f85328; no voters: 39981a8dfe94462b9a73ba489af4c714
I20240502 21:27:21.568967 9092 tablet_bootstrap.cc:654] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:21.569007 9078 raft_consensus.cc:2781] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:21.569118 9078 raft_consensus.cc:482] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:21.569249 9078 raft_consensus.cc:3037] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:21.570894 9092 tablet_bootstrap.cc:492] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:21.571113 9092 ts_tablet_manager.cc:1367] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 21:27:21.572014 9092 raft_consensus.cc:348] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:21.572290 9092 raft_consensus.cc:374] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:21.572216 9078 raft_consensus.cc:504] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:21.572454 9092 raft_consensus.cc:729] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:21.572738 9092 consensus_queue.cc:260] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:21.573462 9092 ts_tablet_manager.cc:1398] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:21.573475 7482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e0911f163f4629a4df0249c5eb8860" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:21.573794 7482 raft_consensus.cc:3037] T 93e0911f163f4629a4df0249c5eb8860 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:21.575536 4291 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:21.576133 7482 raft_consensus.cc:2454] T 93e0911f163f4629a4df0249c5eb8860 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:27:21.576682 4770 leader_election.cc:302] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:21.576949 8561 raft_consensus.cc:2781] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:21.577122 8561 raft_consensus.cc:686] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:27:21.577354 8561 consensus_queue.cc:237] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:21.583943 4294 catalog_manager.cc:5484] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:21.587113 9085 tablet.cc:1722] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:21.587657 9085 tablet.cc:1620] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb: MemRowSet was empty: no flush needed.
I20240502 21:27:21.587759 9078 leader_election.cc:288] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:21.587998 9086 tablet.cc:1722] T f24a85c6fde64fdaa228561a48cb6dc4 P 7a736a8ebb164ce0a051c0fccd76c3b8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:21.588503 9086 tablet.cc:1620] T f24a85c6fde64fdaa228561a48cb6dc4 P 7a736a8ebb164ce0a051c0fccd76c3b8: MemRowSet was empty: no flush needed.
I20240502 21:27:21.588793 9087 tablet.cc:1722] T f24a85c6fde64fdaa228561a48cb6dc4 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:21.589386 9087 tablet.cc:1620] T f24a85c6fde64fdaa228561a48cb6dc4 P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:21.592308 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e0911f163f4629a4df0249c5eb8860" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:21.592586 7214 raft_consensus.cc:3037] T 93e0911f163f4629a4df0249c5eb8860 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:21.598115 7214 raft_consensus.cc:2454] T 93e0911f163f4629a4df0249c5eb8860 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:27:21.598938 4300 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=176c8f5bcca5418fa720abe7cf080321] alter complete (version 2)
I20240502 21:27:21.633152 9068 raft_consensus.cc:482] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:21.633311 9068 raft_consensus.cc:504] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:21.633920 9068 leader_election.cc:288] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:21.634368 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e66998ede944a6b00478270029ebcf" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:27:21.634402 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e66998ede944a6b00478270029ebcf" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:21.634613 4440 raft_consensus.cc:2454] T 94e66998ede944a6b00478270029ebcf P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:21.634651 7349 raft_consensus.cc:2454] T 94e66998ede944a6b00478270029ebcf P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:21.635080 7148 leader_election.cc:302] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:21.635392 9068 raft_consensus.cc:2781] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:21.635529 9068 raft_consensus.cc:482] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:21.635645 9068 raft_consensus.cc:3037] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:21.637622 9068 raft_consensus.cc:504] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:21.638083 9068 leader_election.cc:288] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:21.638473 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e66998ede944a6b00478270029ebcf" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:21.638499 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e66998ede944a6b00478270029ebcf" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:27:21.638664 4440 raft_consensus.cc:3037] T 94e66998ede944a6b00478270029ebcf P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:21.638681 7349 raft_consensus.cc:3037] T 94e66998ede944a6b00478270029ebcf P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:21.640537 4440 raft_consensus.cc:2454] T 94e66998ede944a6b00478270029ebcf P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:21.640810 7349 raft_consensus.cc:2454] T 94e66998ede944a6b00478270029ebcf P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:21.640919 7148 leader_election.cc:302] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:21.641301 9068 raft_consensus.cc:2781] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:21.643801 9068 raft_consensus.cc:686] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:27:21.644066 9068 consensus_queue.cc:237] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:21.647984 4294 catalog_manager.cc:5484] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69). New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:21.649937 8852 raft_consensus.cc:482] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:21.650149 8852 raft_consensus.cc:504] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:21.650871 8852 leader_election.cc:288] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:21.653437 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20b714b400824ecf86890f92aeff19be" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:27:21.653724 4839 raft_consensus.cc:2454] T 20b714b400824ecf86890f92aeff19be P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:21.654264 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20b714b400824ecf86890f92aeff19be" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:27:21.654351 7416 leader_election.cc:302] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:21.654671 8852 raft_consensus.cc:2781] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:21.654769 8852 raft_consensus.cc:482] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:21.654875 8852 raft_consensus.cc:3037] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:21.657159 8852 raft_consensus.cc:504] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:21.657734 8852 leader_election.cc:288] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:21.658128 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20b714b400824ecf86890f92aeff19be" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:27:21.658373 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20b714b400824ecf86890f92aeff19be" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:27:21.658629 4839 raft_consensus.cc:3037] T 20b714b400824ecf86890f92aeff19be P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:21.658881 4440 raft_consensus.cc:2454] T 20b714b400824ecf86890f92aeff19be P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:21.662083 4839 raft_consensus.cc:2454] T 20b714b400824ecf86890f92aeff19be P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:21.663318 7416 leader_election.cc:302] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:21.663724 8852 raft_consensus.cc:2781] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:21.664160 8852 raft_consensus.cc:686] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Becoming Leader. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Running, Role: LEADER
I20240502 21:27:21.664454 8852 consensus_queue.cc:237] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:21.668211 4294 catalog_manager.cc:5484] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a72286397c4edc9a63eca3412be168 (127.29.82.71). New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:27:21.704375 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:21.711410 9085 tablet.cc:1722] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:21.712031 9086 tablet.cc:1722] T f24a85c6fde64fdaa228561a48cb6dc4 P 7a736a8ebb164ce0a051c0fccd76c3b8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:21.713111 9087 tablet.cc:1722] T f24a85c6fde64fdaa228561a48cb6dc4 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:21.725383 4294 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=176c8f5bcca5418fa720abe7cf080321] alter complete (version 3)
I20240502 21:27:21.830811 4294 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:21.831101 4294 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:21.834584 4294 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet f24a85c6fde64fdaa228561a48cb6dc4
I20240502 21:27:21.835515 7328 tablet_service.cc:1514] Processing DeleteTablet for tablet f24a85c6fde64fdaa228561a48cb6dc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:21 UTC) from {username='slave'} at 127.0.0.1:47078
I20240502 21:27:21.836210 9105 tablet_replica.cc:330] T f24a85c6fde64fdaa228561a48cb6dc4 P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:27:21.836205 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet f24a85c6fde64fdaa228561a48cb6dc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:21 UTC) from {username='slave'} at 127.0.0.1:44764
I20240502 21:27:21.836469 9105 raft_consensus.cc:2229] T f24a85c6fde64fdaa228561a48cb6dc4 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:21.836730 9105 raft_consensus.cc:2258] T f24a85c6fde64fdaa228561a48cb6dc4 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:21.838544 9106 tablet_replica.cc:330] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:27:21.838822 9106 raft_consensus.cc:2229] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:21.839329 9106 raft_consensus.cc:2258] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:21.839345 9105 ts_tablet_manager.cc:1875] T f24a85c6fde64fdaa228561a48cb6dc4 P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:21.839761 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet f24a85c6fde64fdaa228561a48cb6dc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:21 UTC) from {username='slave'} at 127.0.0.1:60018
I20240502 21:27:21.840271 9106 ts_tablet_manager.cc:1875] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:21.846364 9105 ts_tablet_manager.cc:1888] T f24a85c6fde64fdaa228561a48cb6dc4 P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:21.846565 9105 log.cc:1196] T f24a85c6fde64fdaa228561a48cb6dc4 P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/f24a85c6fde64fdaa228561a48cb6dc4
I20240502 21:27:21.847113 9105 ts_tablet_manager.cc:1909] T f24a85c6fde64fdaa228561a48cb6dc4 P d0456c5e55a542128688904e003d403c: Deleting consensus metadata
I20240502 21:27:21.848578 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet f24a85c6fde64fdaa228561a48cb6dc4 (table rebalancer_dynamic_table_new_name [id=176c8f5bcca5418fa720abe7cf080321]) successfully deleted
I20240502 21:27:21.850886 9106 ts_tablet_manager.cc:1888] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:21.851043 9106 log.cc:1196] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/f24a85c6fde64fdaa228561a48cb6dc4
I20240502 21:27:21.852981 9106 ts_tablet_manager.cc:1909] T f24a85c6fde64fdaa228561a48cb6dc4 P 398c3c8f3195492c900439ad5e1931bb: Deleting consensus metadata
I20240502 21:27:21.853072 9107 tablet_replica.cc:330] T f24a85c6fde64fdaa228561a48cb6dc4 P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:27:21.853338 9107 raft_consensus.cc:2229] T f24a85c6fde64fdaa228561a48cb6dc4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:21.853657 9107 raft_consensus.cc:2258] T f24a85c6fde64fdaa228561a48cb6dc4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:21.854097 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet f24a85c6fde64fdaa228561a48cb6dc4 (table rebalancer_dynamic_table_new_name [id=176c8f5bcca5418fa720abe7cf080321]) successfully deleted
I20240502 21:27:21.854655 9107 ts_tablet_manager.cc:1875] T f24a85c6fde64fdaa228561a48cb6dc4 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:21.860430 9107 ts_tablet_manager.cc:1888] T f24a85c6fde64fdaa228561a48cb6dc4 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:21.860556 9107 log.cc:1196] T f24a85c6fde64fdaa228561a48cb6dc4 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/f24a85c6fde64fdaa228561a48cb6dc4
I20240502 21:27:21.861059 9107 ts_tablet_manager.cc:1909] T f24a85c6fde64fdaa228561a48cb6dc4 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting consensus metadata
I20240502 21:27:21.862164 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet f24a85c6fde64fdaa228561a48cb6dc4 (table rebalancer_dynamic_table_new_name [id=176c8f5bcca5418fa720abe7cf080321]) successfully deleted
I20240502 21:27:21.936954 4294 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:21.944901 4422 tablet_service.cc:1467] Processing CreateTablet for tablet e506320622ed441f8bd81a752d1fb0da (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dde0e713affc4c4f906997e18a8a5df5]), partition=
I20240502 21:27:21.945331 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e506320622ed441f8bd81a752d1fb0da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:21.945400 7597 tablet_service.cc:1467] Processing CreateTablet for tablet e506320622ed441f8bd81a752d1fb0da (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dde0e713affc4c4f906997e18a8a5df5]), partition=
I20240502 21:27:21.945791 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e506320622ed441f8bd81a752d1fb0da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:21.950722 9090 tablet_bootstrap.cc:492] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:21.951339 4555 tablet_service.cc:1467] Processing CreateTablet for tablet e506320622ed441f8bd81a752d1fb0da (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dde0e713affc4c4f906997e18a8a5df5]), partition=
I20240502 21:27:21.951733 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e506320622ed441f8bd81a752d1fb0da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:21.952283 9090 tablet_bootstrap.cc:654] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:21.953011 9108 tablet_bootstrap.cc:492] T e506320622ed441f8bd81a752d1fb0da P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:21.955544 9108 tablet_bootstrap.cc:654] T e506320622ed441f8bd81a752d1fb0da P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:21.958050 9108 tablet_bootstrap.cc:492] T e506320622ed441f8bd81a752d1fb0da P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:21.958209 9108 ts_tablet_manager.cc:1367] T e506320622ed441f8bd81a752d1fb0da P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:27:21.958961 9108 raft_consensus.cc:348] T e506320622ed441f8bd81a752d1fb0da P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:21.959197 9108 raft_consensus.cc:374] T e506320622ed441f8bd81a752d1fb0da P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:21.959331 9108 raft_consensus.cc:729] T e506320622ed441f8bd81a752d1fb0da P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:21.959606 9108 consensus_queue.cc:260] T e506320622ed441f8bd81a752d1fb0da P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:21.960249 9108 ts_tablet_manager.cc:1398] T e506320622ed441f8bd81a752d1fb0da P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:21.962383 9090 tablet_bootstrap.cc:492] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:21.962599 9090 ts_tablet_manager.cc:1367] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.010s
I20240502 21:27:21.963497 9090 raft_consensus.cc:348] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:21.963811 9090 raft_consensus.cc:374] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:21.963903 9090 raft_consensus.cc:729] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:21.964171 9090 consensus_queue.cc:260] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:21.964702 9111 tablet_bootstrap.cc:492] T e506320622ed441f8bd81a752d1fb0da P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:21.964850 9090 ts_tablet_manager.cc:1398] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:21.966364 9111 tablet_bootstrap.cc:654] T e506320622ed441f8bd81a752d1fb0da P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:21.973014 9111 tablet_bootstrap.cc:492] T e506320622ed441f8bd81a752d1fb0da P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:21.973206 9111 ts_tablet_manager.cc:1367] T e506320622ed441f8bd81a752d1fb0da P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240502 21:27:21.973974 9111 raft_consensus.cc:348] T e506320622ed441f8bd81a752d1fb0da P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:21.974226 9111 raft_consensus.cc:374] T e506320622ed441f8bd81a752d1fb0da P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:21.974334 9111 raft_consensus.cc:729] T e506320622ed441f8bd81a752d1fb0da P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:21.974577 9111 consensus_queue.cc:260] T e506320622ed441f8bd81a752d1fb0da P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:21.975168 9111 ts_tablet_manager.cc:1398] T e506320622ed441f8bd81a752d1fb0da P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:21.995134 8561 consensus_queue.cc:1035] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:22.014221 8561 consensus_queue.cc:1035] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:22.053647 9068 consensus_queue.cc:1035] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:22.058998 9102 consensus_queue.cc:1035] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:22.108677 9119 raft_consensus.cc:482] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:22.108847 9119 raft_consensus.cc:504] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:22.109504 9119 leader_election.cc:288] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:27:22.110031 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e506320622ed441f8bd81a752d1fb0da" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:22.110028 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e506320622ed441f8bd81a752d1fb0da" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:27:22.110271 4574 raft_consensus.cc:2454] T e506320622ed441f8bd81a752d1fb0da P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:22.110275 7618 raft_consensus.cc:2454] T e506320622ed441f8bd81a752d1fb0da P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:22.110708 4376 leader_election.cc:302] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:22.111001 9119 raft_consensus.cc:2781] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:22.111075 9119 raft_consensus.cc:482] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:22.111128 9119 raft_consensus.cc:3037] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:22.113574 9119 raft_consensus.cc:504] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:22.114529 9119 leader_election.cc:288] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:27:22.115038 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e506320622ed441f8bd81a752d1fb0da" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:22.115125 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e506320622ed441f8bd81a752d1fb0da" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:27:22.115288 7618 raft_consensus.cc:3037] T e506320622ed441f8bd81a752d1fb0da P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:22.115340 4574 raft_consensus.cc:3037] T e506320622ed441f8bd81a752d1fb0da P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:22.118242 7618 raft_consensus.cc:2454] T e506320622ed441f8bd81a752d1fb0da P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:22.118249 4574 raft_consensus.cc:2454] T e506320622ed441f8bd81a752d1fb0da P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:22.118670 4378 leader_election.cc:302] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:27:22.118945 9119 raft_consensus.cc:2781] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:22.119093 9119 raft_consensus.cc:686] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:27:22.119334 9119 consensus_queue.cc:237] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:22.123219 4294 catalog_manager.cc:5484] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:22.179596 4294 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:22.188689 4555 tablet_service.cc:1467] Processing CreateTablet for tablet e9f6f71aaffa41e281e67fec134a4827 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=93af5a5216354f909ee81e7cc68d1068]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:22.189102 7462 tablet_service.cc:1467] Processing CreateTablet for tablet e9f6f71aaffa41e281e67fec134a4827 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=93af5a5216354f909ee81e7cc68d1068]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:22.189254 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9f6f71aaffa41e281e67fec134a4827. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:22.189599 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9f6f71aaffa41e281e67fec134a4827. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:22.190552 7328 tablet_service.cc:1467] Processing CreateTablet for tablet 5d710d1340514764b0c7d38e6bf19436 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=93af5a5216354f909ee81e7cc68d1068]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:22.190913 4686 tablet_service.cc:1467] Processing CreateTablet for tablet e9f6f71aaffa41e281e67fec134a4827 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=93af5a5216354f909ee81e7cc68d1068]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:22.191068 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d710d1340514764b0c7d38e6bf19436. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:22.191347 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9f6f71aaffa41e281e67fec134a4827. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:22.196029 9111 tablet_bootstrap.cc:492] T e9f6f71aaffa41e281e67fec134a4827 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:22.196044 7597 tablet_service.cc:1467] Processing CreateTablet for tablet cfceb76b56924833bd7e6b3901573ee7 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=93af5a5216354f909ee81e7cc68d1068]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:22.196501 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfceb76b56924833bd7e6b3901573ee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:22.197702 9111 tablet_bootstrap.cc:654] T e9f6f71aaffa41e281e67fec134a4827 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:22.198148 4422 tablet_service.cc:1467] Processing CreateTablet for tablet 5d710d1340514764b0c7d38e6bf19436 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=93af5a5216354f909ee81e7cc68d1068]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:22.198468 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d710d1340514764b0c7d38e6bf19436. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:22.198961 4553 tablet_service.cc:1467] Processing CreateTablet for tablet 5d710d1340514764b0c7d38e6bf19436 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=93af5a5216354f909ee81e7cc68d1068]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:22.199347 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d710d1340514764b0c7d38e6bf19436. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:22.201814 4685 tablet_service.cc:1467] Processing CreateTablet for tablet cfceb76b56924833bd7e6b3901573ee7 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=93af5a5216354f909ee81e7cc68d1068]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:22.202283 4685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfceb76b56924833bd7e6b3901573ee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:22.206676 9108 tablet_bootstrap.cc:492] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:22.208201 7195 tablet_service.cc:1467] Processing CreateTablet for tablet cfceb76b56924833bd7e6b3901573ee7 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=93af5a5216354f909ee81e7cc68d1068]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:22.208634 9108 tablet_bootstrap.cc:654] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:22.208755 7195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfceb76b56924833bd7e6b3901573ee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:22.212517 9127 tablet_bootstrap.cc:492] T 5d710d1340514764b0c7d38e6bf19436 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:22.215446 9127 tablet_bootstrap.cc:654] T 5d710d1340514764b0c7d38e6bf19436 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:22.220726 9111 tablet_bootstrap.cc:492] T e9f6f71aaffa41e281e67fec134a4827 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:22.220999 9111 ts_tablet_manager.cc:1367] T e9f6f71aaffa41e281e67fec134a4827 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20240502 21:27:22.223987 9130 tablet_bootstrap.cc:492] T cfceb76b56924833bd7e6b3901573ee7 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:22.224905 9111 raft_consensus.cc:348] T e9f6f71aaffa41e281e67fec134a4827 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:22.225250 9111 raft_consensus.cc:374] T e9f6f71aaffa41e281e67fec134a4827 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:22.225405 9111 raft_consensus.cc:729] T e9f6f71aaffa41e281e67fec134a4827 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:22.225759 9111 consensus_queue.cc:260] T e9f6f71aaffa41e281e67fec134a4827 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:22.226305 4442 raft_consensus.cc:3037] T 20b714b400824ecf86890f92aeff19be P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:22.226380 9130 tablet_bootstrap.cc:654] T cfceb76b56924833bd7e6b3901573ee7 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:22.226966 9111 ts_tablet_manager.cc:1398] T e9f6f71aaffa41e281e67fec134a4827 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 21:27:22.227756 9126 tablet_bootstrap.cc:492] T e9f6f71aaffa41e281e67fec134a4827 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:22.228257 9111 tablet_bootstrap.cc:492] T 5d710d1340514764b0c7d38e6bf19436 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:22.229550 9126 tablet_bootstrap.cc:654] T e9f6f71aaffa41e281e67fec134a4827 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:22.234145 9111 tablet_bootstrap.cc:654] T 5d710d1340514764b0c7d38e6bf19436 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:22.237584 8852 consensus_queue.cc:1035] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:22.237959 9090 tablet_bootstrap.cc:492] T 5d710d1340514764b0c7d38e6bf19436 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:22.239326 9130 tablet_bootstrap.cc:492] T cfceb76b56924833bd7e6b3901573ee7 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:22.239496 9090 tablet_bootstrap.cc:654] T 5d710d1340514764b0c7d38e6bf19436 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:22.239634 9130 ts_tablet_manager.cc:1367] T cfceb76b56924833bd7e6b3901573ee7 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.003s
I20240502 21:27:22.240798 9130 raft_consensus.cc:348] T cfceb76b56924833bd7e6b3901573ee7 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:22.241180 9130 raft_consensus.cc:374] T cfceb76b56924833bd7e6b3901573ee7 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:22.241349 9130 raft_consensus.cc:729] T cfceb76b56924833bd7e6b3901573ee7 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:22.241747 9130 consensus_queue.cc:260] T cfceb76b56924833bd7e6b3901573ee7 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:22.244971 9130 ts_tablet_manager.cc:1398] T cfceb76b56924833bd7e6b3901573ee7 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:27:22.252234 9090 tablet_bootstrap.cc:492] T 5d710d1340514764b0c7d38e6bf19436 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:22.252540 9090 ts_tablet_manager.cc:1367] T 5d710d1340514764b0c7d38e6bf19436 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.002s
I20240502 21:27:22.254053 9090 raft_consensus.cc:348] T 5d710d1340514764b0c7d38e6bf19436 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:22.254331 9111 tablet_bootstrap.cc:492] T 5d710d1340514764b0c7d38e6bf19436 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:22.254451 9090 raft_consensus.cc:374] T 5d710d1340514764b0c7d38e6bf19436 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:22.254539 9111 ts_tablet_manager.cc:1367] T 5d710d1340514764b0c7d38e6bf19436 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.006s
I20240502 21:27:22.254580 9090 raft_consensus.cc:729] T 5d710d1340514764b0c7d38e6bf19436 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:22.254999 9090 consensus_queue.cc:260] T 5d710d1340514764b0c7d38e6bf19436 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:22.255560 9111 raft_consensus.cc:348] T 5d710d1340514764b0c7d38e6bf19436 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:22.255868 9111 raft_consensus.cc:374] T 5d710d1340514764b0c7d38e6bf19436 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:22.255983 9111 raft_consensus.cc:729] T 5d710d1340514764b0c7d38e6bf19436 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:22.256176 9090 ts_tablet_manager.cc:1398] T 5d710d1340514764b0c7d38e6bf19436 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:22.257881 9125 tablet_bootstrap.cc:492] T e9f6f71aaffa41e281e67fec134a4827 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:22.260648 9125 tablet_bootstrap.cc:654] T e9f6f71aaffa41e281e67fec134a4827 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:22.263332 4292 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:22.270370 8852 consensus_queue.cc:1035] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:22.280831 9111 consensus_queue.cc:260] T 5d710d1340514764b0c7d38e6bf19436 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:22.282824 9111 ts_tablet_manager.cc:1398] T 5d710d1340514764b0c7d38e6bf19436 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.028s user 0.003s sys 0.000s
I20240502 21:27:22.283265 9126 tablet_bootstrap.cc:492] T e9f6f71aaffa41e281e67fec134a4827 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:27:22.283428 9126 ts_tablet_manager.cc:1367] T e9f6f71aaffa41e281e67fec134a4827 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.056s user 0.014s sys 0.019s
I20240502 21:27:22.284539 9126 raft_consensus.cc:348] T e9f6f71aaffa41e281e67fec134a4827 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:22.284951 9126 raft_consensus.cc:374] T e9f6f71aaffa41e281e67fec134a4827 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:22.285128 9126 raft_consensus.cc:729] T e9f6f71aaffa41e281e67fec134a4827 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:22.285493 9126 consensus_queue.cc:260] T e9f6f71aaffa41e281e67fec134a4827 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:22.286301 9126 ts_tablet_manager.cc:1398] T e9f6f71aaffa41e281e67fec134a4827 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:22.286629 9126 tablet_bootstrap.cc:492] T cfceb76b56924833bd7e6b3901573ee7 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:22.286808 9108 tablet_bootstrap.cc:492] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:22.287065 9108 ts_tablet_manager.cc:1367] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.080s user 0.001s sys 0.011s
I20240502 21:27:22.288735 9126 tablet_bootstrap.cc:654] T cfceb76b56924833bd7e6b3901573ee7 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:22.290159 9108 raft_consensus.cc:348] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:22.290499 9108 raft_consensus.cc:374] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:22.290639 9108 raft_consensus.cc:729] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:22.291730 9108 consensus_queue.cc:260] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:22.295432 9108 ts_tablet_manager.cc:1398] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20240502 21:27:22.302080 4574 raft_consensus.cc:1261] T e506320622ed441f8bd81a752d1fb0da P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:22.306651 9119 consensus_queue.cc:1035] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:22.313746 7618 raft_consensus.cc:1261] T e506320622ed441f8bd81a752d1fb0da P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:22.329109 9127 tablet_bootstrap.cc:492] T 5d710d1340514764b0c7d38e6bf19436 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:22.329386 9127 ts_tablet_manager.cc:1367] T 5d710d1340514764b0c7d38e6bf19436 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.117s user 0.010s sys 0.003s
I20240502 21:27:22.330394 9127 raft_consensus.cc:348] T 5d710d1340514764b0c7d38e6bf19436 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:22.330684 9127 raft_consensus.cc:374] T 5d710d1340514764b0c7d38e6bf19436 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:22.330844 9127 raft_consensus.cc:729] T 5d710d1340514764b0c7d38e6bf19436 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:22.331171 9127 consensus_queue.cc:260] T 5d710d1340514764b0c7d38e6bf19436 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:22.331846 9127 ts_tablet_manager.cc:1398] T 5d710d1340514764b0c7d38e6bf19436 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:22.334321 9119 consensus_queue.cc:1035] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:22.352739 4300 catalog_manager.cc:5576] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781) does not have the latest schema for tablet e506320622ed441f8bd81a752d1fb0da (table rebalancer_dynamic_table [id=dde0e713affc4c4f906997e18a8a5df5]). Expected version 1 got 0
I20240502 21:27:22.372881 4300 catalog_manager.cc:5576] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223) does not have the latest schema for tablet e506320622ed441f8bd81a752d1fb0da (table rebalancer_dynamic_table [id=dde0e713affc4c4f906997e18a8a5df5]). Expected version 1 got 0
I20240502 21:27:22.378268 9147 tablet.cc:1722] T e506320622ed441f8bd81a752d1fb0da P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:22.378954 9147 tablet.cc:1620] T e506320622ed441f8bd81a752d1fb0da P 0ddd486109f5434cace13d60d633ac04: MemRowSet was empty: no flush needed.
I20240502 21:27:22.382428 4300 catalog_manager.cc:5576] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223) does not have the latest schema for tablet e506320622ed441f8bd81a752d1fb0da (table rebalancer_dynamic_table [id=dde0e713affc4c4f906997e18a8a5df5]). Expected version 1 got 0
I20240502 21:27:22.403986 4300 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=dde0e713affc4c4f906997e18a8a5df5] alter complete (version 1)
I20240502 21:27:22.419167 4300 catalog_manager.cc:5576] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781) does not have the latest schema for tablet e506320622ed441f8bd81a752d1fb0da (table rebalancer_dynamic_table [id=dde0e713affc4c4f906997e18a8a5df5]). Expected version 1 got 0
I20240502 21:27:22.422263 9148 tablet.cc:1722] T e506320622ed441f8bd81a752d1fb0da P 7a736a8ebb164ce0a051c0fccd76c3b8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:22.425614 9148 tablet.cc:1620] T e506320622ed441f8bd81a752d1fb0da P 7a736a8ebb164ce0a051c0fccd76c3b8: MemRowSet was empty: no flush needed.
I20240502 21:27:22.427950 9146 tablet.cc:1722] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:22.428925 9146 tablet.cc:1620] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:22.440071 9125 tablet_bootstrap.cc:492] T e9f6f71aaffa41e281e67fec134a4827 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:22.440121 9132 raft_consensus.cc:482] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:22.440263 9125 ts_tablet_manager.cc:1367] T e9f6f71aaffa41e281e67fec134a4827 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.184s user 0.006s sys 0.005s
I20240502 21:27:22.440282 9132 raft_consensus.cc:504] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:22.441124 9125 raft_consensus.cc:348] T e9f6f71aaffa41e281e67fec134a4827 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:22.441256 9132 leader_election.cc:288] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:22.441479 9125 raft_consensus.cc:374] T e9f6f71aaffa41e281e67fec134a4827 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:22.441605 9125 raft_consensus.cc:729] T e9f6f71aaffa41e281e67fec134a4827 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:22.441923 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfceb76b56924833bd7e6b3901573ee7" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:22.441931 9125 consensus_queue.cc:260] T e9f6f71aaffa41e281e67fec134a4827 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:22.442167 7214 raft_consensus.cc:2454] T cfceb76b56924833bd7e6b3901573ee7 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:22.442701 9125 ts_tablet_manager.cc:1398] T e9f6f71aaffa41e281e67fec134a4827 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:22.442904 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfceb76b56924833bd7e6b3901573ee7" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:27:22.443500 7551 leader_election.cc:302] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 39981a8dfe94462b9a73ba489af4c714; no voters:
W20240502 21:27:22.444054 7552 leader_election.cc:341] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:22.445408 9132 raft_consensus.cc:2781] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:22.445515 9132 raft_consensus.cc:482] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:22.445585 9132 raft_consensus.cc:3037] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:22.448635 9126 tablet_bootstrap.cc:492] T cfceb76b56924833bd7e6b3901573ee7 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:27:22.448884 9126 ts_tablet_manager.cc:1367] T cfceb76b56924833bd7e6b3901573ee7 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.162s user 0.046s sys 0.053s
I20240502 21:27:22.449669 9126 raft_consensus.cc:348] T cfceb76b56924833bd7e6b3901573ee7 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:22.449836 9126 raft_consensus.cc:374] T cfceb76b56924833bd7e6b3901573ee7 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:22.449883 9126 raft_consensus.cc:729] T cfceb76b56924833bd7e6b3901573ee7 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:22.450098 9126 consensus_queue.cc:260] T cfceb76b56924833bd7e6b3901573ee7 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:22.450588 9126 ts_tablet_manager.cc:1398] T cfceb76b56924833bd7e6b3901573ee7 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:22.453369 9132 raft_consensus.cc:504] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:22.453567 9119 raft_consensus.cc:482] T 5d710d1340514764b0c7d38e6bf19436 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:22.453719 9119 raft_consensus.cc:504] T 5d710d1340514764b0c7d38e6bf19436 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:22.454370 9119 leader_election.cc:288] T 5d710d1340514764b0c7d38e6bf19436 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:27:22.454833 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfceb76b56924833bd7e6b3901573ee7" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:27:22.455008 4705 raft_consensus.cc:3037] T cfceb76b56924833bd7e6b3901573ee7 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:22.454959 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d710d1340514764b0c7d38e6bf19436" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:27:22.455009 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d710d1340514764b0c7d38e6bf19436" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:22.455216 7350 raft_consensus.cc:2454] T 5d710d1340514764b0c7d38e6bf19436 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:22.455215 4574 raft_consensus.cc:2454] T 5d710d1340514764b0c7d38e6bf19436 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:22.455685 4378 leader_election.cc:302] T 5d710d1340514764b0c7d38e6bf19436 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:22.457233 9119 raft_consensus.cc:2781] T 5d710d1340514764b0c7d38e6bf19436 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:22.457293 9119 raft_consensus.cc:482] T 5d710d1340514764b0c7d38e6bf19436 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:22.457343 9119 raft_consensus.cc:3037] T 5d710d1340514764b0c7d38e6bf19436 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:22.457849 4705 raft_consensus.cc:2454] T cfceb76b56924833bd7e6b3901573ee7 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:22.458388 7552 leader_election.cc:302] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 398c3c8f3195492c900439ad5e1931bb; no voters:
I20240502 21:27:22.461571 9071 raft_consensus.cc:2781] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:22.463838 9119 raft_consensus.cc:504] T 5d710d1340514764b0c7d38e6bf19436 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:22.464807 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d710d1340514764b0c7d38e6bf19436" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:22.464897 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfceb76b56924833bd7e6b3901573ee7" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:22.464996 7350 raft_consensus.cc:3037] T 5d710d1340514764b0c7d38e6bf19436 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:22.465104 7214 raft_consensus.cc:3037] T cfceb76b56924833bd7e6b3901573ee7 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:22.465972 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d710d1340514764b0c7d38e6bf19436" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:27:22.466205 4574 raft_consensus.cc:3037] T 5d710d1340514764b0c7d38e6bf19436 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:22.468566 7214 raft_consensus.cc:2454] T cfceb76b56924833bd7e6b3901573ee7 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:22.469547 9132 leader_election.cc:288] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:22.475309 7350 raft_consensus.cc:2454] T 5d710d1340514764b0c7d38e6bf19436 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:22.476112 4377 leader_election.cc:302] T 5d710d1340514764b0c7d38e6bf19436 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:22.476382 9128 raft_consensus.cc:2781] T 5d710d1340514764b0c7d38e6bf19436 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:22.476524 9128 raft_consensus.cc:686] T 5d710d1340514764b0c7d38e6bf19436 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:27:22.476835 9128 consensus_queue.cc:237] T 5d710d1340514764b0c7d38e6bf19436 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:22.477283 4574 raft_consensus.cc:2454] T 5d710d1340514764b0c7d38e6bf19436 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:22.464365 9119 leader_election.cc:288] T 5d710d1340514764b0c7d38e6bf19436 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:27:22.507361 9071 raft_consensus.cc:686] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Becoming Leader. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Running, Role: LEADER
I20240502 21:27:22.507822 9071 consensus_queue.cc:237] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:22.511979 4300 catalog_manager.cc:5484] T 5d710d1340514764b0c7d38e6bf19436 P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:22.512459 4292 catalog_manager.cc:5484] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ddd486109f5434cace13d60d633ac04 (127.29.82.72). New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:22.515754 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:22.525627 9146 tablet.cc:1722] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:22.526109 9146 tablet.cc:1620] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:22.526669 9148 tablet.cc:1722] T e506320622ed441f8bd81a752d1fb0da P 7a736a8ebb164ce0a051c0fccd76c3b8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:22.527133 9148 tablet.cc:1620] T e506320622ed441f8bd81a752d1fb0da P 7a736a8ebb164ce0a051c0fccd76c3b8: MemRowSet was empty: no flush needed.
I20240502 21:27:22.540248 9147 tablet.cc:1722] T e506320622ed441f8bd81a752d1fb0da P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:22.540776 9147 tablet.cc:1620] T e506320622ed441f8bd81a752d1fb0da P 0ddd486109f5434cace13d60d633ac04: MemRowSet was empty: no flush needed.
I20240502 21:27:22.551116 4292 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=dde0e713affc4c4f906997e18a8a5df5] alter complete (version 2)
I20240502 21:27:22.554642 9004 raft_consensus.cc:482] T e9f6f71aaffa41e281e67fec134a4827 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:22.554862 9004 raft_consensus.cc:504] T e9f6f71aaffa41e281e67fec134a4827 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:22.556368 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f6f71aaffa41e281e67fec134a4827" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:27:22.556672 4706 raft_consensus.cc:2454] T e9f6f71aaffa41e281e67fec134a4827 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:27:22.556722 9004 leader_election.cc:288] T e9f6f71aaffa41e281e67fec134a4827 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:22.557193 4511 leader_election.cc:302] T e9f6f71aaffa41e281e67fec134a4827 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:27:22.557469 9004 raft_consensus.cc:2781] T e9f6f71aaffa41e281e67fec134a4827 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:22.557603 9004 raft_consensus.cc:482] T e9f6f71aaffa41e281e67fec134a4827 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:22.557710 9004 raft_consensus.cc:3037] T e9f6f71aaffa41e281e67fec134a4827 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:22.557615 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f6f71aaffa41e281e67fec134a4827" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:22.557883 7483 raft_consensus.cc:2454] T e9f6f71aaffa41e281e67fec134a4827 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:27:22.560497 9004 raft_consensus.cc:504] T e9f6f71aaffa41e281e67fec134a4827 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:22.561046 9004 leader_election.cc:288] T e9f6f71aaffa41e281e67fec134a4827 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:22.561409 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f6f71aaffa41e281e67fec134a4827" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:27:22.561525 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f6f71aaffa41e281e67fec134a4827" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:22.561650 4705 raft_consensus.cc:3037] T e9f6f71aaffa41e281e67fec134a4827 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:22.561754 7483 raft_consensus.cc:3037] T e9f6f71aaffa41e281e67fec134a4827 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:22.564590 4705 raft_consensus.cc:2454] T e9f6f71aaffa41e281e67fec134a4827 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:27:22.564644 7483 raft_consensus.cc:2454] T e9f6f71aaffa41e281e67fec134a4827 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:27:22.565062 4511 leader_election.cc:302] T e9f6f71aaffa41e281e67fec134a4827 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:27:22.565358 9004 raft_consensus.cc:2781] T e9f6f71aaffa41e281e67fec134a4827 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:22.565541 9004 raft_consensus.cc:686] T e9f6f71aaffa41e281e67fec134a4827 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:27:22.565876 9004 consensus_queue.cc:237] T e9f6f71aaffa41e281e67fec134a4827 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:22.570358 4291 catalog_manager.cc:5484] T e9f6f71aaffa41e281e67fec134a4827 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:27:22.653577 4291 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:22.661759 9146 tablet.cc:1722] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:22.662663 9147 tablet.cc:1722] T e506320622ed441f8bd81a752d1fb0da P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:22.662729 9148 tablet.cc:1722] T e506320622ed441f8bd81a752d1fb0da P 7a736a8ebb164ce0a051c0fccd76c3b8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:22.672258 4300 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=dde0e713affc4c4f906997e18a8a5df5] alter complete (version 3)
I20240502 21:27:22.776237 4294 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:22.776521 4294 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:22.780315 4294 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet e506320622ed441f8bd81a752d1fb0da
I20240502 21:27:22.781476 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet e506320622ed441f8bd81a752d1fb0da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:22 UTC) from {username='slave'} at 127.0.0.1:60018
I20240502 21:27:22.781544 4422 tablet_service.cc:1514] Processing DeleteTablet for tablet e506320622ed441f8bd81a752d1fb0da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:22 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:22.781782 7597 tablet_service.cc:1514] Processing DeleteTablet for tablet e506320622ed441f8bd81a752d1fb0da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:22 UTC) from {username='slave'} at 127.0.0.1:44444
I20240502 21:27:22.794960 9158 tablet_replica.cc:330] T e506320622ed441f8bd81a752d1fb0da P 0ddd486109f5434cace13d60d633ac04: stopping tablet replica
I20240502 21:27:22.795198 9158 raft_consensus.cc:2229] T e506320622ed441f8bd81a752d1fb0da P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:22.795856 9157 tablet_replica.cc:330] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:22.796031 9156 tablet_replica.cc:330] T e506320622ed441f8bd81a752d1fb0da P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:27:22.796237 9157 raft_consensus.cc:2229] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:22.796289 9156 raft_consensus.cc:2229] T e506320622ed441f8bd81a752d1fb0da P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:22.796384 9158 raft_consensus.cc:2258] T e506320622ed441f8bd81a752d1fb0da P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:22.796541 9156 raft_consensus.cc:2258] T e506320622ed441f8bd81a752d1fb0da P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:22.796753 9157 raft_consensus.cc:2258] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:22.797751 9158 ts_tablet_manager.cc:1875] T e506320622ed441f8bd81a752d1fb0da P 0ddd486109f5434cace13d60d633ac04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:22.798727 9156 ts_tablet_manager.cc:1875] T e506320622ed441f8bd81a752d1fb0da P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:22.805641 9156 ts_tablet_manager.cc:1888] T e506320622ed441f8bd81a752d1fb0da P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:22.805763 9156 log.cc:1196] T e506320622ed441f8bd81a752d1fb0da P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/e506320622ed441f8bd81a752d1fb0da
I20240502 21:27:22.809449 9158 ts_tablet_manager.cc:1888] T e506320622ed441f8bd81a752d1fb0da P 0ddd486109f5434cace13d60d633ac04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:22.810169 9156 ts_tablet_manager.cc:1909] T e506320622ed441f8bd81a752d1fb0da P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting consensus metadata
I20240502 21:27:22.810346 9158 log.cc:1196] T e506320622ed441f8bd81a752d1fb0da P 0ddd486109f5434cace13d60d633ac04: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/wals/e506320622ed441f8bd81a752d1fb0da
I20240502 21:27:22.810858 9158 ts_tablet_manager.cc:1909] T e506320622ed441f8bd81a752d1fb0da P 0ddd486109f5434cace13d60d633ac04: Deleting consensus metadata
I20240502 21:27:22.812467 9157 ts_tablet_manager.cc:1875] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:22.812517 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet e506320622ed441f8bd81a752d1fb0da (table rebalancer_dynamic_table_new_name [id=dde0e713affc4c4f906997e18a8a5df5]) successfully deleted
I20240502 21:27:22.813375 4285 catalog_manager.cc:4830] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): tablet e506320622ed441f8bd81a752d1fb0da (table rebalancer_dynamic_table_new_name [id=dde0e713affc4c4f906997e18a8a5df5]) successfully deleted
I20240502 21:27:22.817018 9157 ts_tablet_manager.cc:1888] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:22.817126 9157 log.cc:1196] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/e506320622ed441f8bd81a752d1fb0da
I20240502 21:27:22.817497 9157 ts_tablet_manager.cc:1909] T e506320622ed441f8bd81a752d1fb0da P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:22.818581 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet e506320622ed441f8bd81a752d1fb0da (table rebalancer_dynamic_table_new_name [id=dde0e713affc4c4f906997e18a8a5df5]) successfully deleted
I20240502 21:27:22.870910 9036 consensus_queue.cc:1035] T 5d710d1340514764b0c7d38e6bf19436 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:22.874919 9036 consensus_queue.cc:1035] T 5d710d1340514764b0c7d38e6bf19436 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:22.883782 4294 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:22.888234 9071 consensus_queue.cc:1035] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:22.896770 7328 tablet_service.cc:1467] Processing CreateTablet for tablet f7d52584217b4379a0c750890135255f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=103db7c56b8343369070cc32b99afa81]), partition=
I20240502 21:27:22.897226 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d52584217b4379a0c750890135255f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:22.897918 4818 tablet_service.cc:1467] Processing CreateTablet for tablet f7d52584217b4379a0c750890135255f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=103db7c56b8343369070cc32b99afa81]), partition=
I20240502 21:27:22.898319 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d52584217b4379a0c750890135255f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:22.899776 9150 consensus_queue.cc:1035] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:22.899935 7462 tablet_service.cc:1467] Processing CreateTablet for tablet f7d52584217b4379a0c750890135255f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=103db7c56b8343369070cc32b99afa81]), partition=
I20240502 21:27:22.900290 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d52584217b4379a0c750890135255f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:22.925221 9125 tablet_bootstrap.cc:492] T f7d52584217b4379a0c750890135255f P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:22.927018 9125 tablet_bootstrap.cc:654] T f7d52584217b4379a0c750890135255f P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:22.941391 9125 tablet_bootstrap.cc:492] T f7d52584217b4379a0c750890135255f P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:22.941726 9125 ts_tablet_manager.cc:1367] T f7d52584217b4379a0c750890135255f P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.001s
I20240502 21:27:22.942677 9125 raft_consensus.cc:348] T f7d52584217b4379a0c750890135255f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:22.942931 9125 raft_consensus.cc:374] T f7d52584217b4379a0c750890135255f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:22.943025 9125 raft_consensus.cc:729] T f7d52584217b4379a0c750890135255f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:22.943322 9125 consensus_queue.cc:260] T f7d52584217b4379a0c750890135255f P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:22.943934 9125 ts_tablet_manager.cc:1398] T f7d52584217b4379a0c750890135255f P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:22.945720 9168 tablet_bootstrap.cc:492] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:22.947312 9168 tablet_bootstrap.cc:654] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:22.949666 9170 tablet_bootstrap.cc:492] T f7d52584217b4379a0c750890135255f P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:22.951241 9168 tablet_bootstrap.cc:492] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:22.951423 9168 ts_tablet_manager.cc:1367] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 21:27:22.952399 9170 tablet_bootstrap.cc:654] T f7d52584217b4379a0c750890135255f P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:22.952288 9168 raft_consensus.cc:348] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:22.952579 9168 raft_consensus.cc:374] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:22.952693 9168 raft_consensus.cc:729] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:22.952955 9168 consensus_queue.cc:260] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:22.953649 9168 ts_tablet_manager.cc:1398] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:22.954411 9170 tablet_bootstrap.cc:492] T f7d52584217b4379a0c750890135255f P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:22.954550 9170 ts_tablet_manager.cc:1367] T f7d52584217b4379a0c750890135255f P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:27:22.955112 9170 raft_consensus.cc:348] T f7d52584217b4379a0c750890135255f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:22.955286 9170 raft_consensus.cc:374] T f7d52584217b4379a0c750890135255f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:22.955411 9170 raft_consensus.cc:729] T f7d52584217b4379a0c750890135255f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:22.955668 9170 consensus_queue.cc:260] T f7d52584217b4379a0c750890135255f P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:22.956182 9170 ts_tablet_manager.cc:1398] T f7d52584217b4379a0c750890135255f P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:23.089613 9004 consensus_queue.cc:1035] T e9f6f71aaffa41e281e67fec134a4827 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:23.095023 9155 consensus_queue.cc:1035] T e9f6f71aaffa41e281e67fec134a4827 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:23.103613 8963 raft_consensus.cc:482] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:23.103801 8963 raft_consensus.cc:504] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:23.104656 8963 leader_election.cc:288] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:23.104982 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d52584217b4379a0c750890135255f" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:23.105194 7483 raft_consensus.cc:2454] T f7d52584217b4379a0c750890135255f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 0.
I20240502 21:27:23.105181 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d52584217b4379a0c750890135255f" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:27:23.105448 4839 raft_consensus.cc:2454] T f7d52584217b4379a0c750890135255f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 0.
I20240502 21:27:23.105885 7281 leader_election.cc:302] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:23.106179 8963 raft_consensus.cc:2781] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:23.106299 8963 raft_consensus.cc:482] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:23.106388 8963 raft_consensus.cc:3037] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:23.109231 8963 raft_consensus.cc:504] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:23.109836 8963 leader_election.cc:288] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Requested vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:23.110234 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d52584217b4379a0c750890135255f" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:23.110297 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d52584217b4379a0c750890135255f" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:27:23.110414 7483 raft_consensus.cc:3037] T f7d52584217b4379a0c750890135255f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:23.110522 4839 raft_consensus.cc:3037] T f7d52584217b4379a0c750890135255f P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:23.112707 4839 raft_consensus.cc:2454] T f7d52584217b4379a0c750890135255f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
I20240502 21:27:23.112874 7483 raft_consensus.cc:2454] T f7d52584217b4379a0c750890135255f P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
I20240502 21:27:23.113194 7281 leader_election.cc:302] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:23.113545 8963 raft_consensus.cc:2781] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:23.113726 8963 raft_consensus.cc:686] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Becoming Leader. State: Replica: d0456c5e55a542128688904e003d403c, State: Running, Role: LEADER
I20240502 21:27:23.113999 8963 consensus_queue.cc:237] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:23.118443 4291 catalog_manager.cc:5484] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c reported cstate change: term changed from 0 to 1, leader changed from <none> to d0456c5e55a542128688904e003d403c (127.29.82.70). New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:23.129266 4291 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:23.138842 4422 tablet_service.cc:1467] Processing CreateTablet for tablet 2fd2328258cf4e2db0808731a86cfcf9 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=ec5b364156ae4e6ebd1fdaca42b585f6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:23.138891 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 2fd2328258cf4e2db0808731a86cfcf9 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=ec5b364156ae4e6ebd1fdaca42b585f6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:23.139240 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fd2328258cf4e2db0808731a86cfcf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.139320 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fd2328258cf4e2db0808731a86cfcf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.143175 7597 tablet_service.cc:1467] Processing CreateTablet for tablet 773593a488a74091ac74b33af07227a6 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=ec5b364156ae4e6ebd1fdaca42b585f6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:23.143651 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 773593a488a74091ac74b33af07227a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.143905 7195 tablet_service.cc:1467] Processing CreateTablet for tablet 2fd2328258cf4e2db0808731a86cfcf9 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=ec5b364156ae4e6ebd1fdaca42b585f6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:23.144322 7195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fd2328258cf4e2db0808731a86cfcf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.144304 4555 tablet_service.cc:1467] Processing CreateTablet for tablet 773593a488a74091ac74b33af07227a6 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=ec5b364156ae4e6ebd1fdaca42b585f6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:23.144724 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 773593a488a74091ac74b33af07227a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.148007 7194 tablet_service.cc:1467] Processing CreateTablet for tablet db24db8916314fdc8d9c8a3b09019604 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=ec5b364156ae4e6ebd1fdaca42b585f6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:23.148432 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db24db8916314fdc8d9c8a3b09019604. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.149297 4818 tablet_service.cc:1467] Processing CreateTablet for tablet 773593a488a74091ac74b33af07227a6 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=ec5b364156ae4e6ebd1fdaca42b585f6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:23.149757 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 773593a488a74091ac74b33af07227a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.149995 4553 tablet_service.cc:1467] Processing CreateTablet for tablet db24db8916314fdc8d9c8a3b09019604 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=ec5b364156ae4e6ebd1fdaca42b585f6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:23.150411 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db24db8916314fdc8d9c8a3b09019604. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.154354 9178 tablet_bootstrap.cc:492] T 2fd2328258cf4e2db0808731a86cfcf9 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:23.156291 9178 tablet_bootstrap.cc:654] T 2fd2328258cf4e2db0808731a86cfcf9 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:23.157738 7328 tablet_service.cc:1467] Processing CreateTablet for tablet db24db8916314fdc8d9c8a3b09019604 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=ec5b364156ae4e6ebd1fdaca42b585f6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:23.158576 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db24db8916314fdc8d9c8a3b09019604. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.160370 9181 tablet_bootstrap.cc:492] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:23.161151 9179 tablet_bootstrap.cc:492] T 773593a488a74091ac74b33af07227a6 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:23.163024 9179 tablet_bootstrap.cc:654] T 773593a488a74091ac74b33af07227a6 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:23.164211 9181 tablet_bootstrap.cc:654] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:23.165128 9178 tablet_bootstrap.cc:492] T 2fd2328258cf4e2db0808731a86cfcf9 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:23.165297 9178 ts_tablet_manager.cc:1367] T 2fd2328258cf4e2db0808731a86cfcf9 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20240502 21:27:23.166004 9178 raft_consensus.cc:348] T 2fd2328258cf4e2db0808731a86cfcf9 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:23.166260 9178 raft_consensus.cc:374] T 2fd2328258cf4e2db0808731a86cfcf9 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:23.166343 9178 raft_consensus.cc:729] T 2fd2328258cf4e2db0808731a86cfcf9 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:23.166663 9178 consensus_queue.cc:260] T 2fd2328258cf4e2db0808731a86cfcf9 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:23.167297 9178 ts_tablet_manager.cc:1398] T 2fd2328258cf4e2db0808731a86cfcf9 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:23.172744 9179 tablet_bootstrap.cc:492] T 773593a488a74091ac74b33af07227a6 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:23.172930 9179 ts_tablet_manager.cc:1367] T 773593a488a74091ac74b33af07227a6 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.006s
I20240502 21:27:23.174197 9168 tablet_bootstrap.cc:492] T db24db8916314fdc8d9c8a3b09019604 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:23.173724 9179 raft_consensus.cc:348] T 773593a488a74091ac74b33af07227a6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:23.175935 9179 raft_consensus.cc:374] T 773593a488a74091ac74b33af07227a6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:23.176060 9179 raft_consensus.cc:729] T 773593a488a74091ac74b33af07227a6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:23.176276 9168 tablet_bootstrap.cc:654] T db24db8916314fdc8d9c8a3b09019604 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:23.176299 9179 consensus_queue.cc:260] T 773593a488a74091ac74b33af07227a6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:23.176882 9179 ts_tablet_manager.cc:1398] T 773593a488a74091ac74b33af07227a6 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 21:27:23.177177 9179 tablet_bootstrap.cc:492] T db24db8916314fdc8d9c8a3b09019604 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:23.178895 9179 tablet_bootstrap.cc:654] T db24db8916314fdc8d9c8a3b09019604 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:23.191748 9170 tablet_bootstrap.cc:492] T 773593a488a74091ac74b33af07227a6 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:23.194191 9170 tablet_bootstrap.cc:654] T 773593a488a74091ac74b33af07227a6 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:23.195196 9168 tablet_bootstrap.cc:492] T db24db8916314fdc8d9c8a3b09019604 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:23.195461 9168 ts_tablet_manager.cc:1367] T db24db8916314fdc8d9c8a3b09019604 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240502 21:27:23.196406 9168 raft_consensus.cc:348] T db24db8916314fdc8d9c8a3b09019604 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:23.196764 9168 raft_consensus.cc:374] T db24db8916314fdc8d9c8a3b09019604 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:23.196905 9168 raft_consensus.cc:729] T db24db8916314fdc8d9c8a3b09019604 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:23.197420 9168 consensus_queue.cc:260] T db24db8916314fdc8d9c8a3b09019604 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:23.198086 9168 ts_tablet_manager.cc:1398] T db24db8916314fdc8d9c8a3b09019604 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:23.199988 9181 tablet_bootstrap.cc:492] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:23.200207 9181 ts_tablet_manager.cc:1367] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.020s
I20240502 21:27:23.201121 9181 raft_consensus.cc:348] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:23.201419 9181 raft_consensus.cc:374] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:23.201581 9181 raft_consensus.cc:729] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:23.203917 9181 consensus_queue.cc:260] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:23.204779 9181 ts_tablet_manager.cc:1398] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:23.206552 9181 tablet_bootstrap.cc:492] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:23.208375 9181 tablet_bootstrap.cc:654] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:23.215471 9181 tablet_bootstrap.cc:492] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:23.215765 9181 ts_tablet_manager.cc:1367] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20240502 21:27:23.216598 9181 raft_consensus.cc:348] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:23.216920 9181 raft_consensus.cc:374] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:23.217027 9181 raft_consensus.cc:729] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:23.217363 9181 consensus_queue.cc:260] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:23.221550 9181 ts_tablet_manager.cc:1398] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 21:27:23.228438 4291 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:23.235453 4839 raft_consensus.cc:1261] T f7d52584217b4379a0c750890135255f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:23.236222 7483 raft_consensus.cc:1261] T f7d52584217b4379a0c750890135255f P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:23.237030 9102 raft_consensus.cc:482] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:23.237236 9102 raft_consensus.cc:504] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:23.237987 9102 leader_election.cc:288] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:23.238492 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db24db8916314fdc8d9c8a3b09019604" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
W20240502 21:27:23.239214 7151 leader_election.cc:341] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:23.239197 9166 consensus_queue.cc:1035] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:23.240334 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db24db8916314fdc8d9c8a3b09019604" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:23.240552 7349 raft_consensus.cc:2454] T db24db8916314fdc8d9c8a3b09019604 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:23.242058 7150 leader_election.cc:302] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, d0456c5e55a542128688904e003d403c; no voters: 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:27:23.242708 9102 raft_consensus.cc:2781] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:23.242774 9190 tablet_bootstrap.cc:492] T 2fd2328258cf4e2db0808731a86cfcf9 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:23.242822 9102 raft_consensus.cc:482] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:23.242944 9102 raft_consensus.cc:3037] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:23.245268 9190 tablet_bootstrap.cc:654] T 2fd2328258cf4e2db0808731a86cfcf9 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:23.245251 9166 consensus_queue.cc:1035] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:23.245841 9170 tablet_bootstrap.cc:492] T 773593a488a74091ac74b33af07227a6 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:23.246196 9170 ts_tablet_manager.cc:1367] T 773593a488a74091ac74b33af07227a6 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.055s user 0.012s sys 0.000s
I20240502 21:27:23.247085 9170 raft_consensus.cc:348] T 773593a488a74091ac74b33af07227a6 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:23.247351 9170 raft_consensus.cc:374] T 773593a488a74091ac74b33af07227a6 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:23.247449 9170 raft_consensus.cc:729] T 773593a488a74091ac74b33af07227a6 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:23.247766 9170 consensus_queue.cc:260] T 773593a488a74091ac74b33af07227a6 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:23.248757 9170 ts_tablet_manager.cc:1398] T 773593a488a74091ac74b33af07227a6 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:23.250658 9182 tablet_bootstrap.cc:492] T 773593a488a74091ac74b33af07227a6 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:23.253175 9102 raft_consensus.cc:504] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:23.255056 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db24db8916314fdc8d9c8a3b09019604" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
W20240502 21:27:23.255810 7151 leader_election.cc:341] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:23.264752 9182 tablet_bootstrap.cc:654] T 773593a488a74091ac74b33af07227a6 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:23.264904 9179 tablet_bootstrap.cc:492] T db24db8916314fdc8d9c8a3b09019604 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:23.265131 9179 ts_tablet_manager.cc:1367] T db24db8916314fdc8d9c8a3b09019604 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.088s user 0.044s sys 0.037s
I20240502 21:27:23.266105 9179 raft_consensus.cc:348] T db24db8916314fdc8d9c8a3b09019604 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:23.266459 9179 raft_consensus.cc:374] T db24db8916314fdc8d9c8a3b09019604 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:23.266593 9179 raft_consensus.cc:729] T db24db8916314fdc8d9c8a3b09019604 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:23.266968 9179 consensus_queue.cc:260] T db24db8916314fdc8d9c8a3b09019604 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:23.267841 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db24db8916314fdc8d9c8a3b09019604" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:23.268009 7349 raft_consensus.cc:3037] T db24db8916314fdc8d9c8a3b09019604 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:23.270238 7349 raft_consensus.cc:2454] T db24db8916314fdc8d9c8a3b09019604 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:23.271181 7150 leader_election.cc:302] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, d0456c5e55a542128688904e003d403c; no voters: 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:27:23.271674 9068 raft_consensus.cc:2781] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:23.272138 9068 raft_consensus.cc:686] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:27:23.272486 9068 consensus_queue.cc:237] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:23.273511 9179 ts_tablet_manager.cc:1398] T db24db8916314fdc8d9c8a3b09019604 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240502 21:27:23.274457 9182 tablet_bootstrap.cc:492] T 773593a488a74091ac74b33af07227a6 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:23.274674 9182 ts_tablet_manager.cc:1367] T 773593a488a74091ac74b33af07227a6 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.000s
I20240502 21:27:23.275640 9182 raft_consensus.cc:348] T 773593a488a74091ac74b33af07227a6 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:23.276206 9182 raft_consensus.cc:374] T 773593a488a74091ac74b33af07227a6 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:23.276327 9182 raft_consensus.cc:729] T 773593a488a74091ac74b33af07227a6 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:23.276569 9182 consensus_queue.cc:260] T 773593a488a74091ac74b33af07227a6 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:23.267611 9102 leader_election.cc:288] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:23.277096 4300 catalog_manager.cc:5484] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69). New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:23.278976 9190 tablet_bootstrap.cc:492] T 2fd2328258cf4e2db0808731a86cfcf9 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:27:23.279174 9190 ts_tablet_manager.cc:1367] T 2fd2328258cf4e2db0808731a86cfcf9 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.037s user 0.008s sys 0.000s
I20240502 21:27:23.280066 9190 raft_consensus.cc:348] T 2fd2328258cf4e2db0808731a86cfcf9 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:23.280722 9190 raft_consensus.cc:374] T 2fd2328258cf4e2db0808731a86cfcf9 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:23.280851 9190 raft_consensus.cc:729] T 2fd2328258cf4e2db0808731a86cfcf9 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:23.281095 9190 consensus_queue.cc:260] T 2fd2328258cf4e2db0808731a86cfcf9 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:23.281936 9190 ts_tablet_manager.cc:1398] T 2fd2328258cf4e2db0808731a86cfcf9 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:27:23.282407 9182 ts_tablet_manager.cc:1398] T 773593a488a74091ac74b33af07227a6 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 21:27:23.289695 9102 raft_consensus.cc:482] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:23.289880 9102 raft_consensus.cc:504] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:23.292342 9102 leader_election.cc:288] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:23.293074 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd2328258cf4e2db0808731a86cfcf9" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:27:23.293361 4705 raft_consensus.cc:2454] T 2fd2328258cf4e2db0808731a86cfcf9 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:23.293742 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd2328258cf4e2db0808731a86cfcf9" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:27:23.293987 4442 raft_consensus.cc:2454] T 2fd2328258cf4e2db0808731a86cfcf9 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:23.294463 7148 leader_election.cc:302] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:23.294821 9102 raft_consensus.cc:2781] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:23.294931 9102 raft_consensus.cc:482] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:23.295054 9102 raft_consensus.cc:3037] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:23.298437 9102 raft_consensus.cc:504] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:23.299258 9102 leader_election.cc:288] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:23.300596 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd2328258cf4e2db0808731a86cfcf9" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:27:23.300782 4442 raft_consensus.cc:3037] T 2fd2328258cf4e2db0808731a86cfcf9 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:23.306223 4300 catalog_manager.cc:5576] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913) does not have the latest schema for tablet f7d52584217b4379a0c750890135255f (table rebalancer_dynamic_table [id=103db7c56b8343369070cc32b99afa81]). Expected version 1 got 0
I20240502 21:27:23.308873 9004 raft_consensus.cc:482] T 773593a488a74091ac74b33af07227a6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:23.309047 9004 raft_consensus.cc:504] T 773593a488a74091ac74b33af07227a6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:23.309336 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd2328258cf4e2db0808731a86cfcf9" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:27:23.309581 4706 raft_consensus.cc:3037] T 2fd2328258cf4e2db0808731a86cfcf9 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:23.309803 9004 leader_election.cc:288] T 773593a488a74091ac74b33af07227a6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:23.310745 4837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "773593a488a74091ac74b33af07227a6" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:27:23.310978 4837 raft_consensus.cc:2454] T 773593a488a74091ac74b33af07227a6 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:27:23.311398 4508 leader_election.cc:302] T 773593a488a74091ac74b33af07227a6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:27:23.311717 9004 raft_consensus.cc:2781] T 773593a488a74091ac74b33af07227a6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:23.311834 9004 raft_consensus.cc:482] T 773593a488a74091ac74b33af07227a6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:23.311950 9004 raft_consensus.cc:3037] T 773593a488a74091ac74b33af07227a6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:23.311879 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "773593a488a74091ac74b33af07227a6" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:23.312158 7618 raft_consensus.cc:2454] T 773593a488a74091ac74b33af07227a6 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:27:23.315768 4706 raft_consensus.cc:2454] T 2fd2328258cf4e2db0808731a86cfcf9 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:23.315753 9004 raft_consensus.cc:504] T 773593a488a74091ac74b33af07227a6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:23.316251 7151 leader_election.cc:302] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:23.316615 9068 raft_consensus.cc:2781] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:23.316781 9068 raft_consensus.cc:686] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:27:23.316745 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "773593a488a74091ac74b33af07227a6" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:23.316931 7618 raft_consensus.cc:3037] T 773593a488a74091ac74b33af07227a6 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:23.317059 9068 consensus_queue.cc:237] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:23.320509 7618 raft_consensus.cc:2454] T 773593a488a74091ac74b33af07227a6 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:27:23.320969 4509 leader_election.cc:302] T 773593a488a74091ac74b33af07227a6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:27:23.321272 9155 raft_consensus.cc:2781] T 773593a488a74091ac74b33af07227a6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:23.321527 4442 raft_consensus.cc:2454] T 2fd2328258cf4e2db0808731a86cfcf9 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:23.323732 9004 leader_election.cc:288] T 773593a488a74091ac74b33af07227a6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Requested vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:23.323756 4837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "773593a488a74091ac74b33af07227a6" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:27:23.324018 4837 raft_consensus.cc:3037] T 773593a488a74091ac74b33af07227a6 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:23.327596 4837 raft_consensus.cc:2454] T 773593a488a74091ac74b33af07227a6 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:27:23.330128 9155 raft_consensus.cc:686] T 773593a488a74091ac74b33af07227a6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:27:23.332283 9155 consensus_queue.cc:237] T 773593a488a74091ac74b33af07227a6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:23.353273 4300 catalog_manager.cc:5484] T 773593a488a74091ac74b33af07227a6 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:23.353745 9199 tablet.cc:1722] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:23.361325 9199 tablet.cc:1620] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:23.361436 4300 catalog_manager.cc:5484] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69). New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:23.370029 4300 catalog_manager.cc:5576] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449) does not have the latest schema for tablet f7d52584217b4379a0c750890135255f (table rebalancer_dynamic_table [id=103db7c56b8343369070cc32b99afa81]). Expected version 1 got 0
I20240502 21:27:23.383620 4294 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=103db7c56b8343369070cc32b99afa81] alter complete (version 1)
I20240502 21:27:23.390125 9204 tablet.cc:1722] T f7d52584217b4379a0c750890135255f P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:23.390836 9203 tablet.cc:1722] T f7d52584217b4379a0c750890135255f P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:23.391779 9203 tablet.cc:1620] T f7d52584217b4379a0c750890135255f P 73a1f2ca1f544de9b0e3b37015f85328: MemRowSet was empty: no flush needed.
I20240502 21:27:23.392740 4300 catalog_manager.cc:5576] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913) does not have the latest schema for tablet f7d52584217b4379a0c750890135255f (table rebalancer_dynamic_table [id=103db7c56b8343369070cc32b99afa81]). Expected version 1 got 0
I20240502 21:27:23.393662 4294 catalog_manager.cc:5576] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449) does not have the latest schema for tablet f7d52584217b4379a0c750890135255f (table rebalancer_dynamic_table [id=103db7c56b8343369070cc32b99afa81]). Expected version 1 got 0
I20240502 21:27:23.401129 9204 tablet.cc:1620] T f7d52584217b4379a0c750890135255f P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:23.486071 4300 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:23.494313 9199 tablet.cc:1722] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:23.494832 9199 tablet.cc:1620] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:23.494859 9203 tablet.cc:1722] T f7d52584217b4379a0c750890135255f P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:23.495172 9204 tablet.cc:1722] T f7d52584217b4379a0c750890135255f P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:23.495620 9203 tablet.cc:1620] T f7d52584217b4379a0c750890135255f P 73a1f2ca1f544de9b0e3b37015f85328: MemRowSet was empty: no flush needed.
I20240502 21:27:23.495697 9204 tablet.cc:1620] T f7d52584217b4379a0c750890135255f P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:23.507192 4300 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=103db7c56b8343369070cc32b99afa81] alter complete (version 2)
I20240502 21:27:23.609356 4300 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:23.616983 9199 tablet.cc:1722] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:23.617702 9204 tablet.cc:1722] T f7d52584217b4379a0c750890135255f P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:23.617985 9203 tablet.cc:1722] T f7d52584217b4379a0c750890135255f P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:23.627667 4291 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=103db7c56b8343369070cc32b99afa81] alter complete (version 3)
I20240502 21:27:23.729307 4294 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:23.729593 4294 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:23.733000 4294 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet f7d52584217b4379a0c750890135255f
I20240502 21:27:23.733805 7462 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d52584217b4379a0c750890135255f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:23 UTC) from {username='slave'} at 127.0.0.1:37182
I20240502 21:27:23.733872 4818 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d52584217b4379a0c750890135255f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:23 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:27:23.734493 9205 tablet_replica.cc:330] T f7d52584217b4379a0c750890135255f P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:27:23.734560 9206 tablet_replica.cc:330] T f7d52584217b4379a0c750890135255f P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:27:23.734714 9205 raft_consensus.cc:2229] T f7d52584217b4379a0c750890135255f P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:23.734807 9206 raft_consensus.cc:2229] T f7d52584217b4379a0c750890135255f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:23.734984 9205 raft_consensus.cc:2258] T f7d52584217b4379a0c750890135255f P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:23.735065 9206 raft_consensus.cc:2258] T f7d52584217b4379a0c750890135255f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:23.735818 9205 ts_tablet_manager.cc:1875] T f7d52584217b4379a0c750890135255f P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:23.736227 9206 ts_tablet_manager.cc:1875] T f7d52584217b4379a0c750890135255f P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:23.737174 7328 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d52584217b4379a0c750890135255f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:23 UTC) from {username='slave'} at 127.0.0.1:47078
I20240502 21:27:23.737773 9207 tablet_replica.cc:330] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:27:23.738036 9207 raft_consensus.cc:2229] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:23.738507 9207 raft_consensus.cc:2258] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:23.739445 9207 ts_tablet_manager.cc:1875] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:23.741961 9206 ts_tablet_manager.cc:1888] T f7d52584217b4379a0c750890135255f P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:23.742143 9206 log.cc:1196] T f7d52584217b4379a0c750890135255f P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/f7d52584217b4379a0c750890135255f
I20240502 21:27:23.742260 9205 ts_tablet_manager.cc:1888] T f7d52584217b4379a0c750890135255f P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:23.742724 9206 ts_tablet_manager.cc:1909] T f7d52584217b4379a0c750890135255f P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:27:23.742715 9205 log.cc:1196] T f7d52584217b4379a0c750890135255f P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/f7d52584217b4379a0c750890135255f
I20240502 21:27:23.743180 9205 ts_tablet_manager.cc:1909] T f7d52584217b4379a0c750890135255f P 31a72286397c4edc9a63eca3412be168: Deleting consensus metadata
I20240502 21:27:23.743978 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet f7d52584217b4379a0c750890135255f (table rebalancer_dynamic_table_new_name [id=103db7c56b8343369070cc32b99afa81]) successfully deleted
I20240502 21:27:23.744994 9201 raft_consensus.cc:482] T db24db8916314fdc8d9c8a3b09019604 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:23.745178 9201 raft_consensus.cc:504] T db24db8916314fdc8d9c8a3b09019604 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:23.745925 9207 ts_tablet_manager.cc:1888] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:23.746009 9201 leader_election.cc:288] T db24db8916314fdc8d9c8a3b09019604 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:23.746165 9207 log.cc:1196] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/f7d52584217b4379a0c750890135255f
I20240502 21:27:23.746678 9207 ts_tablet_manager.cc:1909] T f7d52584217b4379a0c750890135255f P d0456c5e55a542128688904e003d403c: Deleting consensus metadata
I20240502 21:27:23.746672 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db24db8916314fdc8d9c8a3b09019604" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:23.746969 7349 raft_consensus.cc:2379] T db24db8916314fdc8d9c8a3b09019604 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in current term 1: Already voted for candidate 39981a8dfe94462b9a73ba489af4c714 in this term.
I20240502 21:27:23.747349 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db24db8916314fdc8d9c8a3b09019604" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:23.747938 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet f7d52584217b4379a0c750890135255f (table rebalancer_dynamic_table_new_name [id=103db7c56b8343369070cc32b99afa81]) successfully deleted
I20240502 21:27:23.748179 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet f7d52584217b4379a0c750890135255f (table rebalancer_dynamic_table_new_name [id=103db7c56b8343369070cc32b99afa81]) successfully deleted
I20240502 21:27:23.748476 4510 leader_election.cc:302] T db24db8916314fdc8d9c8a3b09019604 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a736a8ebb164ce0a051c0fccd76c3b8; no voters: 39981a8dfe94462b9a73ba489af4c714, d0456c5e55a542128688904e003d403c
I20240502 21:27:23.748920 9201 raft_consensus.cc:3037] T db24db8916314fdc8d9c8a3b09019604 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:23.751850 9201 raft_consensus.cc:2726] T db24db8916314fdc8d9c8a3b09019604 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:27:23.783604 9195 consensus_queue.cc:1035] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:23.790851 9102 consensus_queue.cc:1035] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:23.800941 9102 consensus_queue.cc:1035] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:23.839707 9068 consensus_queue.cc:1035] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:23.849742 4294 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:23.871285 7328 tablet_service.cc:1467] Processing CreateTablet for tablet 2954df3fbc7a416c97d9794abfd41b1b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7cc4a72cdc3442f78fc848c030d88055]), partition=
I20240502 21:27:23.871968 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2954df3fbc7a416c97d9794abfd41b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.874815 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 2954df3fbc7a416c97d9794abfd41b1b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7cc4a72cdc3442f78fc848c030d88055]), partition=
I20240502 21:27:23.875193 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2954df3fbc7a416c97d9794abfd41b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.875600 4422 tablet_service.cc:1467] Processing CreateTablet for tablet 2954df3fbc7a416c97d9794abfd41b1b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7cc4a72cdc3442f78fc848c030d88055]), partition=
I20240502 21:27:23.875958 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2954df3fbc7a416c97d9794abfd41b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.882812 9217 tablet_bootstrap.cc:492] T 2954df3fbc7a416c97d9794abfd41b1b P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:23.884639 9217 tablet_bootstrap.cc:654] T 2954df3fbc7a416c97d9794abfd41b1b P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:23.885520 9155 consensus_queue.cc:1035] T 773593a488a74091ac74b33af07227a6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:23.894697 9216 tablet_bootstrap.cc:492] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:23.896536 9217 tablet_bootstrap.cc:492] T 2954df3fbc7a416c97d9794abfd41b1b P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:27:23.896781 9217 ts_tablet_manager.cc:1367] T 2954df3fbc7a416c97d9794abfd41b1b P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20240502 21:27:23.897028 9216 tablet_bootstrap.cc:654] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:23.897874 9217 raft_consensus.cc:348] T 2954df3fbc7a416c97d9794abfd41b1b P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:23.898388 9217 raft_consensus.cc:374] T 2954df3fbc7a416c97d9794abfd41b1b P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:23.898517 9217 raft_consensus.cc:729] T 2954df3fbc7a416c97d9794abfd41b1b P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:23.898764 9217 consensus_queue.cc:260] T 2954df3fbc7a416c97d9794abfd41b1b P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:23.899914 9217 ts_tablet_manager.cc:1398] T 2954df3fbc7a416c97d9794abfd41b1b P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:27:23.902098 9216 tablet_bootstrap.cc:492] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:23.902313 9216 ts_tablet_manager.cc:1367] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240502 21:27:23.903082 4291 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:23.908425 9216 raft_consensus.cc:348] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:23.908758 9216 raft_consensus.cc:374] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:23.908917 9216 raft_consensus.cc:729] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:23.909384 9216 consensus_queue.cc:260] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:23.912623 9216 ts_tablet_manager.cc:1398] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 21:27:23.918071 4686 tablet_service.cc:1467] Processing CreateTablet for tablet fed543e42f844227a14a8d812ce158ba (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=9ce8c844ead645739f9a918d2d899852]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:23.918581 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fed543e42f844227a14a8d812ce158ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.920645 7462 tablet_service.cc:1467] Processing CreateTablet for tablet fed543e42f844227a14a8d812ce158ba (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=9ce8c844ead645739f9a918d2d899852]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:23.921224 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fed543e42f844227a14a8d812ce158ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.925769 7328 tablet_service.cc:1467] Processing CreateTablet for tablet fed543e42f844227a14a8d812ce158ba (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=9ce8c844ead645739f9a918d2d899852]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:23.925863 4421 tablet_service.cc:1467] Processing CreateTablet for tablet 9e6feb60ae03404e85ef76499de190c3 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=9ce8c844ead645739f9a918d2d899852]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:23.926235 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fed543e42f844227a14a8d812ce158ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.926277 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e6feb60ae03404e85ef76499de190c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.927474 4818 tablet_service.cc:1467] Processing CreateTablet for tablet 448ac663d66e4b008bfd9a6c2213fdc2 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=9ce8c844ead645739f9a918d2d899852]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:23.927965 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 448ac663d66e4b008bfd9a6c2213fdc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.930094 7329 tablet_service.cc:1467] Processing CreateTablet for tablet 448ac663d66e4b008bfd9a6c2213fdc2 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=9ce8c844ead645739f9a918d2d899852]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:23.930470 7329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 448ac663d66e4b008bfd9a6c2213fdc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.935506 7463 tablet_service.cc:1467] Processing CreateTablet for tablet 448ac663d66e4b008bfd9a6c2213fdc2 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=9ce8c844ead645739f9a918d2d899852]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:23.935957 7463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 448ac663d66e4b008bfd9a6c2213fdc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.938318 9201 consensus_queue.cc:1035] T 773593a488a74091ac74b33af07227a6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:23.940289 9217 tablet_bootstrap.cc:492] T fed543e42f844227a14a8d812ce158ba P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:23.942128 9217 tablet_bootstrap.cc:654] T fed543e42f844227a14a8d812ce158ba P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:23.943858 7597 tablet_service.cc:1467] Processing CreateTablet for tablet 9e6feb60ae03404e85ef76499de190c3 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=9ce8c844ead645739f9a918d2d899852]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:23.944370 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e6feb60ae03404e85ef76499de190c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.947274 7194 tablet_service.cc:1467] Processing CreateTablet for tablet 9e6feb60ae03404e85ef76499de190c3 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=9ce8c844ead645739f9a918d2d899852]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:23.947857 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e6feb60ae03404e85ef76499de190c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.965183 9228 tablet_bootstrap.cc:492] T 448ac663d66e4b008bfd9a6c2213fdc2 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:23.967310 9229 tablet_bootstrap.cc:492] T 9e6feb60ae03404e85ef76499de190c3 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:23.970708 9217 tablet_bootstrap.cc:492] T fed543e42f844227a14a8d812ce158ba P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:27:23.971125 9229 tablet_bootstrap.cc:654] T 9e6feb60ae03404e85ef76499de190c3 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:23.973130 9217 ts_tablet_manager.cc:1367] T fed543e42f844227a14a8d812ce158ba P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.003s
I20240502 21:27:23.974550 9217 raft_consensus.cc:348] T fed543e42f844227a14a8d812ce158ba P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:23.977337 9217 raft_consensus.cc:374] T fed543e42f844227a14a8d812ce158ba P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:23.977764 9217 raft_consensus.cc:729] T fed543e42f844227a14a8d812ce158ba P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:23.978433 9227 tablet_bootstrap.cc:492] T 9e6feb60ae03404e85ef76499de190c3 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:23.979041 9217 consensus_queue.cc:260] T fed543e42f844227a14a8d812ce158ba P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:23.981276 9227 tablet_bootstrap.cc:654] T 9e6feb60ae03404e85ef76499de190c3 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:23.983451 9217 ts_tablet_manager.cc:1398] T fed543e42f844227a14a8d812ce158ba P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240502 21:27:23.986778 9228 tablet_bootstrap.cc:654] T 448ac663d66e4b008bfd9a6c2213fdc2 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:23.988773 9222 tablet_bootstrap.cc:492] T 448ac663d66e4b008bfd9a6c2213fdc2 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:23.991204 9222 tablet_bootstrap.cc:654] T 448ac663d66e4b008bfd9a6c2213fdc2 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:23.992967 9198 raft_consensus.cc:482] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:23.993144 9198 raft_consensus.cc:504] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:23.994253 9198 leader_election.cc:288] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:23.995378 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2954df3fbc7a416c97d9794abfd41b1b" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:27:23.995498 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2954df3fbc7a416c97d9794abfd41b1b" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:27:23.995793 4705 raft_consensus.cc:2454] T 2954df3fbc7a416c97d9794abfd41b1b P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 0.
W20240502 21:27:23.996253 7281 leader_election.cc:341] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:23.996577 7285 leader_election.cc:302] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, d0456c5e55a542128688904e003d403c; no voters: 71e1375b799b439fa3c83ca763c039c1
I20240502 21:27:23.997342 9198 raft_consensus.cc:2781] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:23.997473 9198 raft_consensus.cc:482] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:23.997589 9198 raft_consensus.cc:3037] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:27:23.997494 7419 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.70:40719, user_credentials={real_user=slave}} blocked reactor thread for 51123us
I20240502 21:27:24.000716 9198 raft_consensus.cc:504] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:24.001927 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2954df3fbc7a416c97d9794abfd41b1b" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:27:24.002136 4705 raft_consensus.cc:3037] T 2954df3fbc7a416c97d9794abfd41b1b P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:24.005029 4705 raft_consensus.cc:2454] T 2954df3fbc7a416c97d9794abfd41b1b P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
I20240502 21:27:24.005523 7285 leader_election.cc:302] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:24.005820 8963 raft_consensus.cc:2781] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:24.005959 8963 raft_consensus.cc:686] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Becoming Leader. State: Replica: d0456c5e55a542128688904e003d403c, State: Running, Role: LEADER
I20240502 21:27:24.006225 8963 consensus_queue.cc:237] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:24.008862 9198 leader_election.cc:288] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:24.009467 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2954df3fbc7a416c97d9794abfd41b1b" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
W20240502 21:27:24.010355 7281 leader_election.cc:341] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:24.025343 9216 tablet_bootstrap.cc:492] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:24.027791 9216 tablet_bootstrap.cc:654] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:24.032289 9222 tablet_bootstrap.cc:492] T 448ac663d66e4b008bfd9a6c2213fdc2 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:24.035769 9222 ts_tablet_manager.cc:1367] T 448ac663d66e4b008bfd9a6c2213fdc2 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.047s user 0.006s sys 0.001s
I20240502 21:27:24.039232 9221 tablet_bootstrap.cc:492] T 2954df3fbc7a416c97d9794abfd41b1b P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:24.041668 9221 tablet_bootstrap.cc:654] T 2954df3fbc7a416c97d9794abfd41b1b P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:24.043196 9229 tablet_bootstrap.cc:492] T 9e6feb60ae03404e85ef76499de190c3 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:24.044492 9229 ts_tablet_manager.cc:1367] T 9e6feb60ae03404e85ef76499de190c3 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.077s user 0.008s sys 0.000s
I20240502 21:27:24.045480 9229 raft_consensus.cc:348] T 9e6feb60ae03404e85ef76499de190c3 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:24.046496 9228 tablet_bootstrap.cc:492] T 448ac663d66e4b008bfd9a6c2213fdc2 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:24.046728 9228 ts_tablet_manager.cc:1367] T 448ac663d66e4b008bfd9a6c2213fdc2 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.082s user 0.003s sys 0.006s
I20240502 21:27:24.047789 9229 raft_consensus.cc:374] T 9e6feb60ae03404e85ef76499de190c3 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:24.048260 9228 raft_consensus.cc:348] T 448ac663d66e4b008bfd9a6c2213fdc2 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:24.048492 9228 raft_consensus.cc:374] T 448ac663d66e4b008bfd9a6c2213fdc2 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:24.049057 9222 raft_consensus.cc:348] T 448ac663d66e4b008bfd9a6c2213fdc2 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:24.049299 9222 raft_consensus.cc:374] T 448ac663d66e4b008bfd9a6c2213fdc2 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:24.049398 9222 raft_consensus.cc:729] T 448ac663d66e4b008bfd9a6c2213fdc2 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:24.049693 9222 consensus_queue.cc:260] T 448ac663d66e4b008bfd9a6c2213fdc2 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:24.051391 9216 tablet_bootstrap.cc:492] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:24.051645 9216 ts_tablet_manager.cc:1367] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.004s
I20240502 21:27:24.052568 9216 raft_consensus.cc:348] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:24.052891 9216 raft_consensus.cc:374] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:24.053062 9216 raft_consensus.cc:729] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:24.053311 9222 ts_tablet_manager.cc:1398] T 448ac663d66e4b008bfd9a6c2213fdc2 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240502 21:27:24.053607 4291 catalog_manager.cc:5484] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c reported cstate change: term changed from 0 to 1, leader changed from <none> to d0456c5e55a542128688904e003d403c (127.29.82.70). New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:24.055274 9227 tablet_bootstrap.cc:492] T 9e6feb60ae03404e85ef76499de190c3 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:24.055501 9227 ts_tablet_manager.cc:1367] T 9e6feb60ae03404e85ef76499de190c3 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.077s user 0.006s sys 0.006s
I20240502 21:27:24.059760 9227 raft_consensus.cc:348] T 9e6feb60ae03404e85ef76499de190c3 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:24.060066 9227 raft_consensus.cc:374] T 9e6feb60ae03404e85ef76499de190c3 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:24.060179 9227 raft_consensus.cc:729] T 9e6feb60ae03404e85ef76499de190c3 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:24.061574 9216 consensus_queue.cc:260] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:24.062646 9227 consensus_queue.cc:260] T 9e6feb60ae03404e85ef76499de190c3 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:24.063355 9227 ts_tablet_manager.cc:1398] T 9e6feb60ae03404e85ef76499de190c3 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240502 21:27:24.063526 9229 raft_consensus.cc:729] T 9e6feb60ae03404e85ef76499de190c3 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:24.068624 9229 consensus_queue.cc:260] T 9e6feb60ae03404e85ef76499de190c3 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:24.069312 9228 raft_consensus.cc:729] T 448ac663d66e4b008bfd9a6c2213fdc2 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:24.070122 9216 ts_tablet_manager.cc:1398] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.017s user 0.000s sys 0.003s
I20240502 21:27:24.070654 9216 tablet_bootstrap.cc:492] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:24.073099 9216 tablet_bootstrap.cc:654] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:24.074512 9228 consensus_queue.cc:260] T 448ac663d66e4b008bfd9a6c2213fdc2 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:24.075455 9228 ts_tablet_manager.cc:1398] T 448ac663d66e4b008bfd9a6c2213fdc2 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.029s user 0.000s sys 0.001s
I20240502 21:27:24.075897 9228 tablet_bootstrap.cc:492] T fed543e42f844227a14a8d812ce158ba P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:24.078212 9228 tablet_bootstrap.cc:654] T fed543e42f844227a14a8d812ce158ba P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:24.082612 9229 ts_tablet_manager.cc:1398] T 9e6feb60ae03404e85ef76499de190c3 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.038s user 0.000s sys 0.002s
I20240502 21:27:24.089258 9221 tablet_bootstrap.cc:492] T 2954df3fbc7a416c97d9794abfd41b1b P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:24.089421 9221 ts_tablet_manager.cc:1367] T 2954df3fbc7a416c97d9794abfd41b1b P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.050s user 0.018s sys 0.025s
I20240502 21:27:24.090133 9221 raft_consensus.cc:348] T 2954df3fbc7a416c97d9794abfd41b1b P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:24.090333 9221 raft_consensus.cc:374] T 2954df3fbc7a416c97d9794abfd41b1b P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:24.090381 9221 raft_consensus.cc:729] T 2954df3fbc7a416c97d9794abfd41b1b P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:24.090600 9221 consensus_queue.cc:260] T 2954df3fbc7a416c97d9794abfd41b1b P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:24.092399 9221 ts_tablet_manager.cc:1398] T 2954df3fbc7a416c97d9794abfd41b1b P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:27:24.092656 9221 tablet_bootstrap.cc:492] T 9e6feb60ae03404e85ef76499de190c3 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:24.094384 9221 tablet_bootstrap.cc:654] T 9e6feb60ae03404e85ef76499de190c3 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:24.097648 9216 tablet_bootstrap.cc:492] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:24.097918 9216 ts_tablet_manager.cc:1367] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20240502 21:27:24.098877 9216 raft_consensus.cc:348] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:24.099648 9216 raft_consensus.cc:374] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:24.100116 9221 tablet_bootstrap.cc:492] T 9e6feb60ae03404e85ef76499de190c3 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:24.100163 9216 raft_consensus.cc:729] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:24.100263 9221 ts_tablet_manager.cc:1367] T 9e6feb60ae03404e85ef76499de190c3 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20240502 21:27:24.100838 9216 consensus_queue.cc:260] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:24.100960 9221 raft_consensus.cc:348] T 9e6feb60ae03404e85ef76499de190c3 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:24.101142 9221 raft_consensus.cc:374] T 9e6feb60ae03404e85ef76499de190c3 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:24.101204 9221 raft_consensus.cc:729] T 9e6feb60ae03404e85ef76499de190c3 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:24.101411 9221 consensus_queue.cc:260] T 9e6feb60ae03404e85ef76499de190c3 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:24.102113 9221 ts_tablet_manager.cc:1398] T 9e6feb60ae03404e85ef76499de190c3 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:24.104209 9216 ts_tablet_manager.cc:1398] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 21:27:24.111219 9228 tablet_bootstrap.cc:492] T fed543e42f844227a14a8d812ce158ba P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:24.111413 9228 ts_tablet_manager.cc:1367] T fed543e42f844227a14a8d812ce158ba P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.036s user 0.009s sys 0.007s
I20240502 21:27:24.112197 9228 raft_consensus.cc:348] T fed543e42f844227a14a8d812ce158ba P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:24.112435 9228 raft_consensus.cc:374] T fed543e42f844227a14a8d812ce158ba P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:24.112506 9228 raft_consensus.cc:729] T fed543e42f844227a14a8d812ce158ba P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:24.112754 9228 consensus_queue.cc:260] T fed543e42f844227a14a8d812ce158ba P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:24.113389 9228 ts_tablet_manager.cc:1398] T fed543e42f844227a14a8d812ce158ba P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:24.167505 4291 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:24.175212 4704 raft_consensus.cc:1261] T 2954df3fbc7a416c97d9794abfd41b1b P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:24.176184 4440 raft_consensus.cc:3037] T 2954df3fbc7a416c97d9794abfd41b1b P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:24.176442 9198 consensus_queue.cc:1035] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:24.179147 4440 raft_consensus.cc:1261] T 2954df3fbc7a416c97d9794abfd41b1b P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:24.179703 9198 consensus_queue.cc:1035] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:24.181284 4291 catalog_manager.cc:5576] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537) does not have the latest schema for tablet 2954df3fbc7a416c97d9794abfd41b1b (table rebalancer_dynamic_table [id=7cc4a72cdc3442f78fc848c030d88055]). Expected version 1 got 0
I20240502 21:27:24.184783 9242 tablet.cc:1722] T 2954df3fbc7a416c97d9794abfd41b1b P 398c3c8f3195492c900439ad5e1931bb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:24.185472 9242 tablet.cc:1620] T 2954df3fbc7a416c97d9794abfd41b1b P 398c3c8f3195492c900439ad5e1931bb: MemRowSet was empty: no flush needed.
I20240502 21:27:24.187616 4291 catalog_manager.cc:5576] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537) does not have the latest schema for tablet 2954df3fbc7a416c97d9794abfd41b1b (table rebalancer_dynamic_table [id=7cc4a72cdc3442f78fc848c030d88055]). Expected version 1 got 0
I20240502 21:27:24.192611 4291 catalog_manager.cc:5576] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781) does not have the latest schema for tablet 2954df3fbc7a416c97d9794abfd41b1b (table rebalancer_dynamic_table [id=7cc4a72cdc3442f78fc848c030d88055]). Expected version 1 got 0
I20240502 21:27:24.195926 9241 tablet.cc:1722] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:24.197342 9241 tablet.cc:1620] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:24.198202 4291 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=7cc4a72cdc3442f78fc848c030d88055] alter complete (version 1)
I20240502 21:27:24.200587 9102 raft_consensus.cc:482] T 9e6feb60ae03404e85ef76499de190c3 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:24.200775 9102 raft_consensus.cc:504] T 9e6feb60ae03404e85ef76499de190c3 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:24.201049 9244 tablet.cc:1722] T 2954df3fbc7a416c97d9794abfd41b1b P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:24.201526 9102 leader_election.cc:288] T 9e6feb60ae03404e85ef76499de190c3 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:24.201762 4294 catalog_manager.cc:5576] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781) does not have the latest schema for tablet 2954df3fbc7a416c97d9794abfd41b1b (table rebalancer_dynamic_table [id=7cc4a72cdc3442f78fc848c030d88055]). Expected version 1 got 0
I20240502 21:27:24.202232 7617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e6feb60ae03404e85ef76499de190c3" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:24.202469 7617 raft_consensus.cc:2454] T 9e6feb60ae03404e85ef76499de190c3 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:24.202817 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e6feb60ae03404e85ef76499de190c3" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:27:24.203013 4442 raft_consensus.cc:2454] T 9e6feb60ae03404e85ef76499de190c3 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 0.
I20240502 21:27:24.203868 7148 leader_election.cc:302] T 9e6feb60ae03404e85ef76499de190c3 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:24.204161 9102 raft_consensus.cc:2781] T 9e6feb60ae03404e85ef76499de190c3 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:24.204281 9102 raft_consensus.cc:482] T 9e6feb60ae03404e85ef76499de190c3 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:24.204396 9102 raft_consensus.cc:3037] T 9e6feb60ae03404e85ef76499de190c3 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:24.206804 9244 tablet.cc:1620] T 2954df3fbc7a416c97d9794abfd41b1b P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:24.207561 9102 raft_consensus.cc:504] T 9e6feb60ae03404e85ef76499de190c3 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:24.208202 9102 leader_election.cc:288] T 9e6feb60ae03404e85ef76499de190c3 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Requested vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:24.208578 7617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e6feb60ae03404e85ef76499de190c3" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:24.208609 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e6feb60ae03404e85ef76499de190c3" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:27:24.208761 7617 raft_consensus.cc:3037] T 9e6feb60ae03404e85ef76499de190c3 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:24.208812 4440 raft_consensus.cc:3037] T 9e6feb60ae03404e85ef76499de190c3 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:24.210999 7617 raft_consensus.cc:2454] T 9e6feb60ae03404e85ef76499de190c3 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:24.211512 7149 leader_election.cc:302] T 9e6feb60ae03404e85ef76499de190c3 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:24.211966 9102 raft_consensus.cc:2781] T 9e6feb60ae03404e85ef76499de190c3 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:24.212155 9102 raft_consensus.cc:686] T 9e6feb60ae03404e85ef76499de190c3 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:27:24.212716 9102 consensus_queue.cc:237] T 9e6feb60ae03404e85ef76499de190c3 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:24.215559 4440 raft_consensus.cc:2454] T 9e6feb60ae03404e85ef76499de190c3 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:27:24.216722 4294 catalog_manager.cc:5484] T 9e6feb60ae03404e85ef76499de190c3 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69). New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:24.254868 9198 raft_consensus.cc:482] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:24.255064 9198 raft_consensus.cc:504] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:24.255815 9198 leader_election.cc:288] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:24.256130 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "448ac663d66e4b008bfd9a6c2213fdc2" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:27:24.256284 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "448ac663d66e4b008bfd9a6c2213fdc2" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:24.256376 4839 raft_consensus.cc:2454] T 448ac663d66e4b008bfd9a6c2213fdc2 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 0.
I20240502 21:27:24.256516 7483 raft_consensus.cc:2454] T 448ac663d66e4b008bfd9a6c2213fdc2 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 0.
I20240502 21:27:24.256850 7281 leader_election.cc:302] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:24.257133 9198 raft_consensus.cc:2781] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:24.257272 9198 raft_consensus.cc:482] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:24.257390 9198 raft_consensus.cc:3037] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:24.259886 9198 raft_consensus.cc:504] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:24.260903 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "448ac663d66e4b008bfd9a6c2213fdc2" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:27:24.261114 4839 raft_consensus.cc:3037] T 448ac663d66e4b008bfd9a6c2213fdc2 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:24.263206 4839 raft_consensus.cc:2454] T 448ac663d66e4b008bfd9a6c2213fdc2 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
I20240502 21:27:24.263617 9198 leader_election.cc:288] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Requested vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:24.263623 7281 leader_election.cc:302] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:24.263952 9198 raft_consensus.cc:2781] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:24.264122 9198 raft_consensus.cc:686] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Becoming Leader. State: Replica: d0456c5e55a542128688904e003d403c, State: Running, Role: LEADER
I20240502 21:27:24.264374 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "448ac663d66e4b008bfd9a6c2213fdc2" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:24.264554 7483 raft_consensus.cc:3037] T 448ac663d66e4b008bfd9a6c2213fdc2 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:24.264427 9198 consensus_queue.cc:237] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:24.266222 9198 raft_consensus.cc:482] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:24.266386 9198 raft_consensus.cc:504] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:24.266971 7483 raft_consensus.cc:2454] T 448ac663d66e4b008bfd9a6c2213fdc2 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
I20240502 21:27:24.267366 7482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fed543e42f844227a14a8d812ce158ba" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:24.267449 4704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fed543e42f844227a14a8d812ce158ba" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:27:24.267614 7482 raft_consensus.cc:2454] T fed543e42f844227a14a8d812ce158ba P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 0.
I20240502 21:27:24.267722 4704 raft_consensus.cc:2454] T fed543e42f844227a14a8d812ce158ba P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 0.
I20240502 21:27:24.267997 9198 leader_election.cc:288] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:27:24.268150 7285 leader_election.cc:302] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:24.268386 9198 raft_consensus.cc:2781] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:24.268464 9198 raft_consensus.cc:482] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:24.268563 9198 raft_consensus.cc:3037] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:24.269098 4294 catalog_manager.cc:5484] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c reported cstate change: term changed from 0 to 1, leader changed from <none> to d0456c5e55a542128688904e003d403c (127.29.82.70). New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:27:24.270610 9198 raft_consensus.cc:504] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:24.271556 9198 leader_election.cc:288] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Requested vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:27:24.271637 7482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fed543e42f844227a14a8d812ce158ba" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:24.271857 7482 raft_consensus.cc:3037] T fed543e42f844227a14a8d812ce158ba P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:24.274047 7482 raft_consensus.cc:2454] T fed543e42f844227a14a8d812ce158ba P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
I20240502 21:27:24.274385 7281 leader_election.cc:302] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:24.274616 9198 raft_consensus.cc:2781] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:24.274863 9198 raft_consensus.cc:686] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Becoming Leader. State: Replica: d0456c5e55a542128688904e003d403c, State: Running, Role: LEADER
I20240502 21:27:24.275120 9198 consensus_queue.cc:237] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:24.275441 4704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fed543e42f844227a14a8d812ce158ba" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:27:24.275681 4704 raft_consensus.cc:3037] T fed543e42f844227a14a8d812ce158ba P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:24.278371 4704 raft_consensus.cc:2454] T fed543e42f844227a14a8d812ce158ba P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
I20240502 21:27:24.279726 4294 catalog_manager.cc:5484] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c reported cstate change: term changed from 0 to 1, leader changed from <none> to d0456c5e55a542128688904e003d403c (127.29.82.70). New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:24.303225 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:24.311385 9241 tablet.cc:1722] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:24.312156 9241 tablet.cc:1620] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:24.312963 9244 tablet.cc:1722] T 2954df3fbc7a416c97d9794abfd41b1b P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:24.313401 9244 tablet.cc:1620] T 2954df3fbc7a416c97d9794abfd41b1b P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:24.314996 9242 tablet.cc:1722] T 2954df3fbc7a416c97d9794abfd41b1b P 398c3c8f3195492c900439ad5e1931bb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:24.315492 9242 tablet.cc:1620] T 2954df3fbc7a416c97d9794abfd41b1b P 398c3c8f3195492c900439ad5e1931bb: MemRowSet was empty: no flush needed.
I20240502 21:27:24.326690 4291 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=7cc4a72cdc3442f78fc848c030d88055] alter complete (version 2)
I20240502 21:27:24.431031 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:24.439565 9241 tablet.cc:1722] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:24.440513 9242 tablet.cc:1722] T 2954df3fbc7a416c97d9794abfd41b1b P 398c3c8f3195492c900439ad5e1931bb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:24.449723 9244 tablet.cc:1722] T 2954df3fbc7a416c97d9794abfd41b1b P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:24.454697 4294 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=7cc4a72cdc3442f78fc848c030d88055] alter complete (version 3)
I20240502 21:27:24.559762 4291 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:24.560038 4291 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:24.563191 4291 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 2954df3fbc7a416c97d9794abfd41b1b
I20240502 21:27:24.564213 4422 tablet_service.cc:1514] Processing DeleteTablet for tablet 2954df3fbc7a416c97d9794abfd41b1b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:24 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:24.564230 7328 tablet_service.cc:1514] Processing DeleteTablet for tablet 2954df3fbc7a416c97d9794abfd41b1b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:24 UTC) from {username='slave'} at 127.0.0.1:47078
I20240502 21:27:24.564218 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet 2954df3fbc7a416c97d9794abfd41b1b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:24 UTC) from {username='slave'} at 127.0.0.1:44764
I20240502 21:27:24.565327 9252 tablet_replica.cc:330] T 2954df3fbc7a416c97d9794abfd41b1b P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:27:24.565455 9251 tablet_replica.cc:330] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:27:24.565646 9252 raft_consensus.cc:2229] T 2954df3fbc7a416c97d9794abfd41b1b P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:24.565708 9251 raft_consensus.cc:2229] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:24.565953 9252 raft_consensus.cc:2258] T 2954df3fbc7a416c97d9794abfd41b1b P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:24.566154 9251 raft_consensus.cc:2258] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:24.567085 9251 ts_tablet_manager.cc:1875] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:24.567173 9252 ts_tablet_manager.cc:1875] T 2954df3fbc7a416c97d9794abfd41b1b P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:24.573621 9251 ts_tablet_manager.cc:1888] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:24.573797 9251 log.cc:1196] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/2954df3fbc7a416c97d9794abfd41b1b
I20240502 21:27:24.574347 9251 ts_tablet_manager.cc:1909] T 2954df3fbc7a416c97d9794abfd41b1b P d0456c5e55a542128688904e003d403c: Deleting consensus metadata
I20240502 21:27:24.574824 9252 ts_tablet_manager.cc:1888] T 2954df3fbc7a416c97d9794abfd41b1b P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:24.574992 9252 log.cc:1196] T 2954df3fbc7a416c97d9794abfd41b1b P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/2954df3fbc7a416c97d9794abfd41b1b
I20240502 21:27:24.575709 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet 2954df3fbc7a416c97d9794abfd41b1b (table rebalancer_dynamic_table_new_name [id=7cc4a72cdc3442f78fc848c030d88055]) successfully deleted
I20240502 21:27:24.576013 9253 tablet_replica.cc:330] T 2954df3fbc7a416c97d9794abfd41b1b P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:24.576274 9253 raft_consensus.cc:2229] T 2954df3fbc7a416c97d9794abfd41b1b P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:24.576421 9252 ts_tablet_manager.cc:1909] T 2954df3fbc7a416c97d9794abfd41b1b P 398c3c8f3195492c900439ad5e1931bb: Deleting consensus metadata
I20240502 21:27:24.576550 9253 raft_consensus.cc:2258] T 2954df3fbc7a416c97d9794abfd41b1b P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:24.577744 9253 ts_tablet_manager.cc:1875] T 2954df3fbc7a416c97d9794abfd41b1b P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:24.580503 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet 2954df3fbc7a416c97d9794abfd41b1b (table rebalancer_dynamic_table_new_name [id=7cc4a72cdc3442f78fc848c030d88055]) successfully deleted
I20240502 21:27:24.582139 9253 ts_tablet_manager.cc:1888] T 2954df3fbc7a416c97d9794abfd41b1b P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:24.582235 9253 log.cc:1196] T 2954df3fbc7a416c97d9794abfd41b1b P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/2954df3fbc7a416c97d9794abfd41b1b
I20240502 21:27:24.582616 9253 ts_tablet_manager.cc:1909] T 2954df3fbc7a416c97d9794abfd41b1b P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:24.583520 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet 2954df3fbc7a416c97d9794abfd41b1b (table rebalancer_dynamic_table_new_name [id=7cc4a72cdc3442f78fc848c030d88055]) successfully deleted
I20240502 21:27:24.610608 9068 consensus_queue.cc:1035] T 9e6feb60ae03404e85ef76499de190c3 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:24.616017 9068 consensus_queue.cc:1035] T 9e6feb60ae03404e85ef76499de190c3 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:24.665905 4292 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:24.674100 4422 tablet_service.cc:1467] Processing CreateTablet for tablet 092f9d4b0b7e4711a883e10e351cb2d7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b1ac1065e1094975ac90e3d2203adea9]), partition=
I20240502 21:27:24.674537 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 092f9d4b0b7e4711a883e10e351cb2d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:24.679528 9258 tablet_bootstrap.cc:492] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:24.679545 7597 tablet_service.cc:1467] Processing CreateTablet for tablet 092f9d4b0b7e4711a883e10e351cb2d7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b1ac1065e1094975ac90e3d2203adea9]), partition=
I20240502 21:27:24.679926 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 092f9d4b0b7e4711a883e10e351cb2d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:24.681190 9258 tablet_bootstrap.cc:654] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:24.682978 9258 tablet_bootstrap.cc:492] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:24.683106 9258 ts_tablet_manager.cc:1367] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 21:27:24.683800 9258 raft_consensus.cc:348] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:24.683952 4818 tablet_service.cc:1467] Processing CreateTablet for tablet 092f9d4b0b7e4711a883e10e351cb2d7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b1ac1065e1094975ac90e3d2203adea9]), partition=
I20240502 21:27:24.684026 9258 raft_consensus.cc:374] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:24.684100 9258 raft_consensus.cc:729] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:24.684324 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 092f9d4b0b7e4711a883e10e351cb2d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:24.684352 9258 consensus_queue.cc:260] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:24.685680 9258 ts_tablet_manager.cc:1398] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:24.695351 9198 consensus_queue.cc:1035] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:24.695813 9260 tablet_bootstrap.cc:492] T 092f9d4b0b7e4711a883e10e351cb2d7 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:24.697762 9260 tablet_bootstrap.cc:654] T 092f9d4b0b7e4711a883e10e351cb2d7 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:24.700239 9260 tablet_bootstrap.cc:492] T 092f9d4b0b7e4711a883e10e351cb2d7 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:24.700454 9260 ts_tablet_manager.cc:1367] T 092f9d4b0b7e4711a883e10e351cb2d7 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 21:27:24.701491 9260 raft_consensus.cc:348] T 092f9d4b0b7e4711a883e10e351cb2d7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:24.701694 9260 raft_consensus.cc:374] T 092f9d4b0b7e4711a883e10e351cb2d7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:24.701824 9260 raft_consensus.cc:729] T 092f9d4b0b7e4711a883e10e351cb2d7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:24.702051 9260 consensus_queue.cc:260] T 092f9d4b0b7e4711a883e10e351cb2d7 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:24.702651 9260 ts_tablet_manager.cc:1398] T 092f9d4b0b7e4711a883e10e351cb2d7 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:24.704155 9262 tablet_bootstrap.cc:492] T 092f9d4b0b7e4711a883e10e351cb2d7 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:24.706202 9262 tablet_bootstrap.cc:654] T 092f9d4b0b7e4711a883e10e351cb2d7 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:24.709180 8963 consensus_queue.cc:1035] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:24.714426 9262 tablet_bootstrap.cc:492] T 092f9d4b0b7e4711a883e10e351cb2d7 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:24.714607 9262 ts_tablet_manager.cc:1367] T 092f9d4b0b7e4711a883e10e351cb2d7 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20240502 21:27:24.715548 9262 raft_consensus.cc:348] T 092f9d4b0b7e4711a883e10e351cb2d7 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:24.715840 9262 raft_consensus.cc:374] T 092f9d4b0b7e4711a883e10e351cb2d7 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:24.715950 9262 raft_consensus.cc:729] T 092f9d4b0b7e4711a883e10e351cb2d7 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:24.716234 9262 consensus_queue.cc:260] T 092f9d4b0b7e4711a883e10e351cb2d7 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:24.716971 9262 ts_tablet_manager.cc:1398] T 092f9d4b0b7e4711a883e10e351cb2d7 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:24.717618 8963 consensus_queue.cc:1035] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:24.723975 9250 consensus_queue.cc:1035] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:24.822729 4294 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:24.842119 7597 tablet_service.cc:1467] Processing CreateTablet for tablet fd95f653fd5f4a038e4bc51dbd72e8e6 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=9e3a4d2e56654ed784b01da7d60bdc59]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:24.842602 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd95f653fd5f4a038e4bc51dbd72e8e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:24.843195 4555 tablet_service.cc:1467] Processing CreateTablet for tablet fd95f653fd5f4a038e4bc51dbd72e8e6 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=9e3a4d2e56654ed784b01da7d60bdc59]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:24.843685 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd95f653fd5f4a038e4bc51dbd72e8e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:24.849097 7194 tablet_service.cc:1467] Processing CreateTablet for tablet fd95f653fd5f4a038e4bc51dbd72e8e6 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=9e3a4d2e56654ed784b01da7d60bdc59]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:24.849557 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd95f653fd5f4a038e4bc51dbd72e8e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:24.849462 7328 tablet_service.cc:1467] Processing CreateTablet for tablet 73413ab7b2514ff8a0f2a9a7bd580b2c (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=9e3a4d2e56654ed784b01da7d60bdc59]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:24.849867 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73413ab7b2514ff8a0f2a9a7bd580b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:24.858245 4553 tablet_service.cc:1467] Processing CreateTablet for tablet 73413ab7b2514ff8a0f2a9a7bd580b2c (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=9e3a4d2e56654ed784b01da7d60bdc59]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:24.858567 4818 tablet_service.cc:1467] Processing CreateTablet for tablet 73413ab7b2514ff8a0f2a9a7bd580b2c (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=9e3a4d2e56654ed784b01da7d60bdc59]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:24.858711 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73413ab7b2514ff8a0f2a9a7bd580b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:24.859009 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73413ab7b2514ff8a0f2a9a7bd580b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:24.861084 4686 tablet_service.cc:1467] Processing CreateTablet for tablet fd610e340cd3409397d8f23b623b4074 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=9e3a4d2e56654ed784b01da7d60bdc59]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:24.861567 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd610e340cd3409397d8f23b623b4074. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:24.861477 4554 tablet_service.cc:1467] Processing CreateTablet for tablet fd610e340cd3409397d8f23b623b4074 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=9e3a4d2e56654ed784b01da7d60bdc59]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:24.861832 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd610e340cd3409397d8f23b623b4074. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:24.869922 7462 tablet_service.cc:1467] Processing CreateTablet for tablet fd610e340cd3409397d8f23b623b4074 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=9e3a4d2e56654ed784b01da7d60bdc59]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:24.870409 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd610e340cd3409397d8f23b623b4074. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:24.870774 9260 tablet_bootstrap.cc:492] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:24.872632 9260 tablet_bootstrap.cc:654] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:24.873746 9271 tablet_bootstrap.cc:492] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:24.875845 9271 tablet_bootstrap.cc:654] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:24.876910 9262 tablet_bootstrap.cc:492] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:24.878934 9262 tablet_bootstrap.cc:654] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:24.885183 9260 tablet_bootstrap.cc:492] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:24.897207 9273 tablet_bootstrap.cc:492] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:24.898138 9260 ts_tablet_manager.cc:1367] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.010s
I20240502 21:27:24.899278 9260 raft_consensus.cc:348] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:24.900563 9273 tablet_bootstrap.cc:654] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:24.900882 9260 raft_consensus.cc:374] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:24.901082 9260 raft_consensus.cc:729] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:24.901527 9260 consensus_queue.cc:260] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:24.902558 9260 ts_tablet_manager.cc:1398] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240502 21:27:24.905841 9271 tablet_bootstrap.cc:492] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:24.906006 9271 ts_tablet_manager.cc:1367] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.011s
I20240502 21:27:24.906849 9271 raft_consensus.cc:348] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:24.907083 9271 raft_consensus.cc:374] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:24.907202 9271 raft_consensus.cc:729] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:24.907439 9271 consensus_queue.cc:260] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:24.908470 9271 ts_tablet_manager.cc:1398] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:24.908730 9271 tablet_bootstrap.cc:492] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:24.910881 9271 tablet_bootstrap.cc:654] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:24.913306 9274 tablet_bootstrap.cc:492] T 73413ab7b2514ff8a0f2a9a7bd580b2c P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:24.915225 9274 tablet_bootstrap.cc:654] T 73413ab7b2514ff8a0f2a9a7bd580b2c P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:24.915722 9278 tablet_bootstrap.cc:492] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:24.916960 9262 tablet_bootstrap.cc:492] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:24.917181 9279 tablet_bootstrap.cc:492] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:24.918097 9262 ts_tablet_manager.cc:1367] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.007s
I20240502 21:27:24.925707 9279 tablet_bootstrap.cc:654] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:24.920871 9262 raft_consensus.cc:348] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:24.927305 9262 raft_consensus.cc:374] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:24.927484 9262 raft_consensus.cc:729] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:24.936782 9278 tablet_bootstrap.cc:654] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:24.936951 9262 consensus_queue.cc:260] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:24.938738 9273 tablet_bootstrap.cc:492] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:24.938966 9273 ts_tablet_manager.cc:1367] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.010s
I20240502 21:27:24.939466 9262 ts_tablet_manager.cc:1398] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.021s user 0.000s sys 0.002s
I20240502 21:27:24.939918 9273 raft_consensus.cc:348] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:24.940217 9273 raft_consensus.cc:374] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:24.940352 9273 raft_consensus.cc:729] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:24.940794 9036 raft_consensus.cc:482] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:24.940943 9036 raft_consensus.cc:504] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:24.941771 9036 leader_election.cc:288] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:24.943004 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "092f9d4b0b7e4711a883e10e351cb2d7" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:27:24.943291 4839 raft_consensus.cc:2454] T 092f9d4b0b7e4711a883e10e351cb2d7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:24.940625 9273 consensus_queue.cc:260] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:24.944069 4375 leader_election.cc:302] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:24.944473 9036 raft_consensus.cc:2781] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:24.944553 9036 raft_consensus.cc:482] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:24.944664 9036 raft_consensus.cc:3037] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:24.945242 9273 ts_tablet_manager.cc:1398] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 21:27:24.947714 7616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "092f9d4b0b7e4711a883e10e351cb2d7" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:24.947966 7616 raft_consensus.cc:2454] T 092f9d4b0b7e4711a883e10e351cb2d7 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:24.958567 9036 raft_consensus.cc:504] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:24.959405 9036 leader_election.cc:288] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:24.959793 7616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "092f9d4b0b7e4711a883e10e351cb2d7" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:24.960112 7616 raft_consensus.cc:3037] T 092f9d4b0b7e4711a883e10e351cb2d7 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:24.960373 9278 tablet_bootstrap.cc:492] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:27:24.960530 9278 ts_tablet_manager.cc:1367] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.002s
I20240502 21:27:24.961612 9271 tablet_bootstrap.cc:492] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:24.961670 9279 tablet_bootstrap.cc:492] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:24.961772 9271 ts_tablet_manager.cc:1367] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.053s user 0.008s sys 0.008s
I20240502 21:27:24.961833 9279 ts_tablet_manager.cc:1367] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.046s user 0.000s sys 0.005s
I20240502 21:27:24.962517 9271 raft_consensus.cc:348] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:24.962739 9271 raft_consensus.cc:374] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:24.962632 9279 raft_consensus.cc:348] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:24.962808 9271 raft_consensus.cc:729] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:24.962888 9279 raft_consensus.cc:374] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:24.962908 7616 raft_consensus.cc:2454] T 092f9d4b0b7e4711a883e10e351cb2d7 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:24.962992 9279 raft_consensus.cc:729] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:24.963022 9271 consensus_queue.cc:260] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:24.963255 9279 consensus_queue.cc:260] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:24.963613 9271 ts_tablet_manager.cc:1398] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:24.963912 9271 tablet_bootstrap.cc:492] T fd610e340cd3409397d8f23b623b4074 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:24.965217 9279 ts_tablet_manager.cc:1398] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240502 21:27:24.965914 9278 raft_consensus.cc:348] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:24.966699 9278 raft_consensus.cc:374] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:24.966847 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "092f9d4b0b7e4711a883e10e351cb2d7" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:27:24.967092 4839 raft_consensus.cc:3037] T 092f9d4b0b7e4711a883e10e351cb2d7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:24.967449 9274 tablet_bootstrap.cc:492] T 73413ab7b2514ff8a0f2a9a7bd580b2c P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:24.967617 9274 ts_tablet_manager.cc:1367] T 73413ab7b2514ff8a0f2a9a7bd580b2c P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.054s user 0.023s sys 0.018s
I20240502 21:27:24.967953 4376 leader_election.cc:302] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:24.968446 9274 raft_consensus.cc:348] T 73413ab7b2514ff8a0f2a9a7bd580b2c P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:24.968600 9249 raft_consensus.cc:2781] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:24.968675 9274 raft_consensus.cc:374] T 73413ab7b2514ff8a0f2a9a7bd580b2c P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:24.968755 9249 raft_consensus.cc:686] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:27:24.968869 9278 raft_consensus.cc:729] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:24.968998 9249 consensus_queue.cc:237] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:24.969089 9278 consensus_queue.cc:260] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:24.969781 9278 ts_tablet_manager.cc:1398] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 21:27:24.969806 4839 raft_consensus.cc:2454] T 092f9d4b0b7e4711a883e10e351cb2d7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:24.972219 9271 tablet_bootstrap.cc:654] T fd610e340cd3409397d8f23b623b4074 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:24.973316 9274 raft_consensus.cc:729] T 73413ab7b2514ff8a0f2a9a7bd580b2c P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:24.979809 9274 consensus_queue.cc:260] T 73413ab7b2514ff8a0f2a9a7bd580b2c P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:24.980643 9274 ts_tablet_manager.cc:1398] T 73413ab7b2514ff8a0f2a9a7bd580b2c P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240502 21:27:24.983148 9271 tablet_bootstrap.cc:492] T fd610e340cd3409397d8f23b623b4074 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:24.983321 9271 ts_tablet_manager.cc:1367] T fd610e340cd3409397d8f23b623b4074 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.006s
I20240502 21:27:24.984484 9271 raft_consensus.cc:348] T fd610e340cd3409397d8f23b623b4074 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:24.984715 9271 raft_consensus.cc:374] T fd610e340cd3409397d8f23b623b4074 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:24.984835 9271 raft_consensus.cc:729] T fd610e340cd3409397d8f23b623b4074 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:24.985131 9271 consensus_queue.cc:260] T fd610e340cd3409397d8f23b623b4074 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:24.987043 9271 ts_tablet_manager.cc:1398] T fd610e340cd3409397d8f23b623b4074 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 21:27:24.988327 4292 catalog_manager.cc:5484] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:25.002802 9272 raft_consensus.cc:482] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:25.002964 9272 raft_consensus.cc:504] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:25.003747 9272 leader_election.cc:288] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:25.004030 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73413ab7b2514ff8a0f2a9a7bd580b2c" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:25.004272 7349 raft_consensus.cc:2454] T 73413ab7b2514ff8a0f2a9a7bd580b2c P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:27:25.004777 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73413ab7b2514ff8a0f2a9a7bd580b2c" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:27:25.005647 4510 leader_election.cc:302] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a736a8ebb164ce0a051c0fccd76c3b8, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:25.006399 9272 raft_consensus.cc:2781] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:25.007112 9272 raft_consensus.cc:482] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:25.007238 9272 raft_consensus.cc:3037] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:25.010438 9272 raft_consensus.cc:504] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:25.011886 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73413ab7b2514ff8a0f2a9a7bd580b2c" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:25.012100 7349 raft_consensus.cc:3037] T 73413ab7b2514ff8a0f2a9a7bd580b2c P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:25.016698 7349 raft_consensus.cc:2454] T 73413ab7b2514ff8a0f2a9a7bd580b2c P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:27:25.017526 4837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73413ab7b2514ff8a0f2a9a7bd580b2c" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:27:25.017728 4837 raft_consensus.cc:3037] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:25.017750 9272 leader_election.cc:288] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:25.019179 4510 leader_election.cc:302] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a736a8ebb164ce0a051c0fccd76c3b8, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:25.019611 9155 raft_consensus.cc:2781] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:25.019814 9155 raft_consensus.cc:686] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:27:25.020139 9155 consensus_queue.cc:237] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:25.020376 4837 raft_consensus.cc:2454] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:27:25.027501 4291 catalog_manager.cc:5484] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:25.101828 4291 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:25.107481 4837 raft_consensus.cc:1261] T 092f9d4b0b7e4711a883e10e351cb2d7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:25.107806 7616 raft_consensus.cc:1261] T 092f9d4b0b7e4711a883e10e351cb2d7 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:25.108119 9128 consensus_queue.cc:1035] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:25.108542 9036 consensus_queue.cc:1035] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:25.114315 4291 catalog_manager.cc:5576] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449) does not have the latest schema for tablet 092f9d4b0b7e4711a883e10e351cb2d7 (table rebalancer_dynamic_table [id=b1ac1065e1094975ac90e3d2203adea9]). Expected version 1 got 0
I20240502 21:27:25.114955 4292 catalog_manager.cc:5576] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223) does not have the latest schema for tablet 092f9d4b0b7e4711a883e10e351cb2d7 (table rebalancer_dynamic_table [id=b1ac1065e1094975ac90e3d2203adea9]). Expected version 1 got 0
I20240502 21:27:25.118834 9291 tablet.cc:1722] T 092f9d4b0b7e4711a883e10e351cb2d7 P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:25.119863 9291 tablet.cc:1620] T 092f9d4b0b7e4711a883e10e351cb2d7 P 73a1f2ca1f544de9b0e3b37015f85328: MemRowSet was empty: no flush needed.
I20240502 21:27:25.122646 9292 tablet.cc:1722] T 092f9d4b0b7e4711a883e10e351cb2d7 P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:25.123998 4292 catalog_manager.cc:5576] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449) does not have the latest schema for tablet 092f9d4b0b7e4711a883e10e351cb2d7 (table rebalancer_dynamic_table [id=b1ac1065e1094975ac90e3d2203adea9]). Expected version 1 got 0
I20240502 21:27:25.124478 9292 tablet.cc:1620] T 092f9d4b0b7e4711a883e10e351cb2d7 P 0ddd486109f5434cace13d60d633ac04: MemRowSet was empty: no flush needed.
I20240502 21:27:25.131402 4292 catalog_manager.cc:5576] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223) does not have the latest schema for tablet 092f9d4b0b7e4711a883e10e351cb2d7 (table rebalancer_dynamic_table [id=b1ac1065e1094975ac90e3d2203adea9]). Expected version 1 got 0
I20240502 21:27:25.135092 9290 tablet.cc:1722] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:25.135675 9290 tablet.cc:1620] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:25.137981 4291 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=b1ac1065e1094975ac90e3d2203adea9] alter complete (version 1)
I20240502 21:27:25.145201 9052 raft_consensus.cc:482] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:25.145397 9052 raft_consensus.cc:504] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:25.146092 9052 leader_election.cc:288] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:27:25.146523 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd610e340cd3409397d8f23b623b4074" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:25.146701 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd610e340cd3409397d8f23b623b4074" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:27:25.146831 7483 raft_consensus.cc:2454] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:27:25.146961 4575 raft_consensus.cc:2454] T fd610e340cd3409397d8f23b623b4074 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:27:25.147262 4639 leader_election.cc:302] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 398c3c8f3195492c900439ad5e1931bb; no voters:
I20240502 21:27:25.147642 9052 raft_consensus.cc:2781] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:25.147734 9052 raft_consensus.cc:482] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:25.147840 9052 raft_consensus.cc:3037] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:25.150087 9052 raft_consensus.cc:504] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:25.150686 9052 leader_election.cc:288] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Requested vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:27:25.150967 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd610e340cd3409397d8f23b623b4074" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:25.151218 7483 raft_consensus.cc:3037] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:25.151170 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd610e340cd3409397d8f23b623b4074" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:27:25.151373 4575 raft_consensus.cc:3037] T fd610e340cd3409397d8f23b623b4074 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:25.154145 7483 raft_consensus.cc:2454] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:27:25.154234 4575 raft_consensus.cc:2454] T fd610e340cd3409397d8f23b623b4074 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:27:25.154587 4642 leader_election.cc:302] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:27:25.154860 9052 raft_consensus.cc:2781] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:25.155037 9052 raft_consensus.cc:686] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Becoming Leader. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Running, Role: LEADER
I20240502 21:27:25.155308 9052 consensus_queue.cc:237] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:25.158705 4292 catalog_manager.cc:5484] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 398c3c8f3195492c900439ad5e1931bb (127.29.82.67). New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:25.170289 9071 raft_consensus.cc:482] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:25.170456 9071 raft_consensus.cc:504] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:25.171268 9071 leader_election.cc:288] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:25.171690 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd95f653fd5f4a038e4bc51dbd72e8e6" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:27:25.171717 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd95f653fd5f4a038e4bc51dbd72e8e6" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:25.171895 4575 raft_consensus.cc:2454] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:25.171919 7214 raft_consensus.cc:2454] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:25.172297 7552 leader_election.cc:302] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 39981a8dfe94462b9a73ba489af4c714, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:27:25.172601 9071 raft_consensus.cc:2781] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:25.172742 9071 raft_consensus.cc:482] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:25.172835 9071 raft_consensus.cc:3037] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:25.174743 9071 raft_consensus.cc:504] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:25.175359 9071 leader_election.cc:288] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Requested vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:25.175906 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd95f653fd5f4a038e4bc51dbd72e8e6" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:27:25.175899 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd95f653fd5f4a038e4bc51dbd72e8e6" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:25.176110 4575 raft_consensus.cc:3037] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:25.176143 7214 raft_consensus.cc:3037] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:25.178956 7214 raft_consensus.cc:2454] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:25.178973 4575 raft_consensus.cc:2454] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:25.179389 7551 leader_election.cc:302] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:25.179729 9071 raft_consensus.cc:2781] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:25.179867 9071 raft_consensus.cc:686] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Becoming Leader. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Running, Role: LEADER
I20240502 21:27:25.180150 9071 consensus_queue.cc:237] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:25.184518 4292 catalog_manager.cc:5484] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ddd486109f5434cace13d60d633ac04 (127.29.82.72). New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:25.245538 4292 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:25.253835 9290 tablet.cc:1722] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:25.254328 9290 tablet.cc:1620] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:25.255110 9291 tablet.cc:1722] T 092f9d4b0b7e4711a883e10e351cb2d7 P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:25.255543 9291 tablet.cc:1620] T 092f9d4b0b7e4711a883e10e351cb2d7 P 73a1f2ca1f544de9b0e3b37015f85328: MemRowSet was empty: no flush needed.
I20240502 21:27:25.256816 9292 tablet.cc:1722] T 092f9d4b0b7e4711a883e10e351cb2d7 P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:25.257385 9292 tablet.cc:1620] T 092f9d4b0b7e4711a883e10e351cb2d7 P 0ddd486109f5434cace13d60d633ac04: MemRowSet was empty: no flush needed.
I20240502 21:27:25.267622 4291 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=b1ac1065e1094975ac90e3d2203adea9] alter complete (version 2)
I20240502 21:27:25.372666 4292 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:25.381274 9290 tablet.cc:1722] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:25.382071 9292 tablet.cc:1722] T 092f9d4b0b7e4711a883e10e351cb2d7 P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:25.396178 9291 tablet.cc:1722] T 092f9d4b0b7e4711a883e10e351cb2d7 P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:25.400103 4292 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=b1ac1065e1094975ac90e3d2203adea9] alter complete (version 3)
I20240502 21:27:25.501386 4292 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:25.501628 4292 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:25.504823 4292 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 092f9d4b0b7e4711a883e10e351cb2d7
I20240502 21:27:25.505678 7597 tablet_service.cc:1514] Processing DeleteTablet for tablet 092f9d4b0b7e4711a883e10e351cb2d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:25 UTC) from {username='slave'} at 127.0.0.1:44444
I20240502 21:27:25.505920 4422 tablet_service.cc:1514] Processing DeleteTablet for tablet 092f9d4b0b7e4711a883e10e351cb2d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:25 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:25.506372 9295 tablet_replica.cc:330] T 092f9d4b0b7e4711a883e10e351cb2d7 P 0ddd486109f5434cace13d60d633ac04: stopping tablet replica
I20240502 21:27:25.507005 9295 raft_consensus.cc:2229] T 092f9d4b0b7e4711a883e10e351cb2d7 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:25.507197 4818 tablet_service.cc:1514] Processing DeleteTablet for tablet 092f9d4b0b7e4711a883e10e351cb2d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:25 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:27:25.507287 9295 raft_consensus.cc:2258] T 092f9d4b0b7e4711a883e10e351cb2d7 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:25.508046 9297 tablet_replica.cc:330] T 092f9d4b0b7e4711a883e10e351cb2d7 P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:27:25.508251 9297 raft_consensus.cc:2229] T 092f9d4b0b7e4711a883e10e351cb2d7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:25.508288 9296 tablet_replica.cc:330] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:25.508430 9297 raft_consensus.cc:2258] T 092f9d4b0b7e4711a883e10e351cb2d7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:25.508500 9296 raft_consensus.cc:2229] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:25.509238 9296 raft_consensus.cc:2258] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:25.509348 9297 ts_tablet_manager.cc:1875] T 092f9d4b0b7e4711a883e10e351cb2d7 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:25.510108 9296 ts_tablet_manager.cc:1875] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:25.510653 9295 ts_tablet_manager.cc:1875] T 092f9d4b0b7e4711a883e10e351cb2d7 P 0ddd486109f5434cace13d60d633ac04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:25.515089 9297 ts_tablet_manager.cc:1888] T 092f9d4b0b7e4711a883e10e351cb2d7 P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:25.515201 9297 log.cc:1196] T 092f9d4b0b7e4711a883e10e351cb2d7 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/092f9d4b0b7e4711a883e10e351cb2d7
I20240502 21:27:25.515617 9297 ts_tablet_manager.cc:1909] T 092f9d4b0b7e4711a883e10e351cb2d7 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:27:25.516808 9296 ts_tablet_manager.cc:1888] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:25.516947 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet 092f9d4b0b7e4711a883e10e351cb2d7 (table rebalancer_dynamic_table_new_name [id=b1ac1065e1094975ac90e3d2203adea9]) successfully deleted
I20240502 21:27:25.517030 9296 log.cc:1196] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/092f9d4b0b7e4711a883e10e351cb2d7
I20240502 21:27:25.517086 9295 ts_tablet_manager.cc:1888] T 092f9d4b0b7e4711a883e10e351cb2d7 P 0ddd486109f5434cace13d60d633ac04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:25.517227 9295 log.cc:1196] T 092f9d4b0b7e4711a883e10e351cb2d7 P 0ddd486109f5434cace13d60d633ac04: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/wals/092f9d4b0b7e4711a883e10e351cb2d7
I20240502 21:27:25.517609 9296 ts_tablet_manager.cc:1909] T 092f9d4b0b7e4711a883e10e351cb2d7 P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:25.517699 9295 ts_tablet_manager.cc:1909] T 092f9d4b0b7e4711a883e10e351cb2d7 P 0ddd486109f5434cace13d60d633ac04: Deleting consensus metadata
I20240502 21:27:25.518764 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet 092f9d4b0b7e4711a883e10e351cb2d7 (table rebalancer_dynamic_table_new_name [id=b1ac1065e1094975ac90e3d2203adea9]) successfully deleted
I20240502 21:27:25.518843 4285 catalog_manager.cc:4830] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): tablet 092f9d4b0b7e4711a883e10e351cb2d7 (table rebalancer_dynamic_table_new_name [id=b1ac1065e1094975ac90e3d2203adea9]) successfully deleted
I20240502 21:27:25.571928 9155 consensus_queue.cc:1035] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:25.577353 9155 consensus_queue.cc:1035] T 73413ab7b2514ff8a0f2a9a7bd580b2c P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:25.584463 9071 consensus_queue.cc:1035] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:25.591640 9150 consensus_queue.cc:1035] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:25.608606 4292 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:25.617002 7462 tablet_service.cc:1467] Processing CreateTablet for tablet a365c229ed9c4c01987b086db659d5f8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=24623a600ef24bddb541ee187c709932]), partition=
I20240502 21:27:25.617403 4686 tablet_service.cc:1467] Processing CreateTablet for tablet a365c229ed9c4c01987b086db659d5f8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=24623a600ef24bddb541ee187c709932]), partition=
I20240502 21:27:25.617518 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a365c229ed9c4c01987b086db659d5f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:25.617547 7328 tablet_service.cc:1467] Processing CreateTablet for tablet a365c229ed9c4c01987b086db659d5f8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=24623a600ef24bddb541ee187c709932]), partition=
I20240502 21:27:25.617779 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a365c229ed9c4c01987b086db659d5f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:25.617899 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a365c229ed9c4c01987b086db659d5f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:25.624193 9173 consensus_queue.cc:1035] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:25.628149 9307 tablet_bootstrap.cc:492] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:25.629070 9305 tablet_bootstrap.cc:492] T a365c229ed9c4c01987b086db659d5f8 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:25.631834 9307 tablet_bootstrap.cc:654] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:25.634142 9305 tablet_bootstrap.cc:654] T a365c229ed9c4c01987b086db659d5f8 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:25.646219 9305 tablet_bootstrap.cc:492] T a365c229ed9c4c01987b086db659d5f8 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:27:25.646546 9305 ts_tablet_manager.cc:1367] T a365c229ed9c4c01987b086db659d5f8 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.008s
I20240502 21:27:25.647760 9305 raft_consensus.cc:348] T a365c229ed9c4c01987b086db659d5f8 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:25.648166 9305 raft_consensus.cc:374] T a365c229ed9c4c01987b086db659d5f8 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:25.648334 9305 raft_consensus.cc:729] T a365c229ed9c4c01987b086db659d5f8 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:25.648694 9305 consensus_queue.cc:260] T a365c229ed9c4c01987b086db659d5f8 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:25.649334 9308 tablet_bootstrap.cc:492] T a365c229ed9c4c01987b086db659d5f8 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:25.651376 9308 tablet_bootstrap.cc:654] T a365c229ed9c4c01987b086db659d5f8 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:25.656128 9305 ts_tablet_manager.cc:1398] T a365c229ed9c4c01987b086db659d5f8 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240502 21:27:25.656709 9052 consensus_queue.cc:1035] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:25.668134 9307 tablet_bootstrap.cc:492] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:25.668303 9307 ts_tablet_manager.cc:1367] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.040s user 0.019s sys 0.004s
I20240502 21:27:25.668550 9308 tablet_bootstrap.cc:492] T a365c229ed9c4c01987b086db659d5f8 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:25.668752 9308 ts_tablet_manager.cc:1367] T a365c229ed9c4c01987b086db659d5f8 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.010s
I20240502 21:27:25.669195 9307 raft_consensus.cc:348] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:25.669440 9307 raft_consensus.cc:374] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:25.669571 9307 raft_consensus.cc:729] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:25.669615 9308 raft_consensus.cc:348] T a365c229ed9c4c01987b086db659d5f8 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:25.669824 9308 raft_consensus.cc:374] T a365c229ed9c4c01987b086db659d5f8 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:25.669926 9308 raft_consensus.cc:729] T a365c229ed9c4c01987b086db659d5f8 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:25.669834 9307 consensus_queue.cc:260] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:25.670150 9308 consensus_queue.cc:260] T a365c229ed9c4c01987b086db659d5f8 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:25.670583 9307 ts_tablet_manager.cc:1398] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:25.670831 9308 ts_tablet_manager.cc:1398] T a365c229ed9c4c01987b086db659d5f8 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:25.695082 4291 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:25.706321 7328 tablet_service.cc:1467] Processing CreateTablet for tablet 180e178d0c714a35b8ca8843fad466c0 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=c758184ce285455e9f74bd2b6fd9dfc7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:25.706418 7597 tablet_service.cc:1467] Processing CreateTablet for tablet 180e178d0c714a35b8ca8843fad466c0 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=c758184ce285455e9f74bd2b6fd9dfc7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:25.706872 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 180e178d0c714a35b8ca8843fad466c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:25.706912 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 180e178d0c714a35b8ca8843fad466c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:25.707226 7194 tablet_service.cc:1467] Processing CreateTablet for tablet 202f8b10d5674afa875da2f3929bd8da (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=c758184ce285455e9f74bd2b6fd9dfc7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:25.707885 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 64aeda4ec20a4b568a475cb380d2568c (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=c758184ce285455e9f74bd2b6fd9dfc7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:25.708176 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 202f8b10d5674afa875da2f3929bd8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:25.708328 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64aeda4ec20a4b568a475cb380d2568c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:25.710640 9198 raft_consensus.cc:482] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:25.710817 9198 raft_consensus.cc:504] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:25.711637 9198 leader_election.cc:288] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:27:25.714687 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a365c229ed9c4c01987b086db659d5f8" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:27:25.715003 4705 raft_consensus.cc:2454] T a365c229ed9c4c01987b086db659d5f8 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 0.
I20240502 21:27:25.716456 7285 leader_election.cc:302] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:25.716958 8963 raft_consensus.cc:2781] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:25.717089 8963 raft_consensus.cc:482] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:25.717209 8963 raft_consensus.cc:3037] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:25.718844 7595 tablet_service.cc:1467] Processing CreateTablet for tablet 202f8b10d5674afa875da2f3929bd8da (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=c758184ce285455e9f74bd2b6fd9dfc7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:25.719471 7595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 202f8b10d5674afa875da2f3929bd8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:25.720744 8963 raft_consensus.cc:504] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:25.723683 9316 tablet_bootstrap.cc:492] T 202f8b10d5674afa875da2f3929bd8da P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:25.724942 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a365c229ed9c4c01987b086db659d5f8" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:27:25.725175 4705 raft_consensus.cc:3037] T a365c229ed9c4c01987b086db659d5f8 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:25.725495 9316 tablet_bootstrap.cc:654] T 202f8b10d5674afa875da2f3929bd8da P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:25.728322 7462 tablet_service.cc:1467] Processing CreateTablet for tablet 180e178d0c714a35b8ca8843fad466c0 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=c758184ce285455e9f74bd2b6fd9dfc7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:25.728809 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 180e178d0c714a35b8ca8843fad466c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:25.729422 4817 tablet_service.cc:1467] Processing CreateTablet for tablet 64aeda4ec20a4b568a475cb380d2568c (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=c758184ce285455e9f74bd2b6fd9dfc7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:25.729928 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64aeda4ec20a4b568a475cb380d2568c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:25.734911 9308 tablet_bootstrap.cc:492] T 180e178d0c714a35b8ca8843fad466c0 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:25.735221 4818 tablet_service.cc:1467] Processing CreateTablet for tablet 202f8b10d5674afa875da2f3929bd8da (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=c758184ce285455e9f74bd2b6fd9dfc7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:25.735620 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 202f8b10d5674afa875da2f3929bd8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:25.735678 9225 raft_consensus.cc:482] T a365c229ed9c4c01987b086db659d5f8 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:25.735841 9225 raft_consensus.cc:504] T a365c229ed9c4c01987b086db659d5f8 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:25.736707 9308 tablet_bootstrap.cc:654] T 180e178d0c714a35b8ca8843fad466c0 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:25.736758 9225 leader_election.cc:288] T a365c229ed9c4c01987b086db659d5f8 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:27:25.737502 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a365c229ed9c4c01987b086db659d5f8" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:25.737784 7350 raft_consensus.cc:2379] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31a72286397c4edc9a63eca3412be168 in current term 1: Already voted for candidate d0456c5e55a542128688904e003d403c in this term.
I20240502 21:27:25.738003 4705 raft_consensus.cc:2454] T a365c229ed9c4c01987b086db659d5f8 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
I20240502 21:27:25.740226 7285 leader_election.cc:302] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:25.740527 8963 leader_election.cc:288] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Requested vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:27:25.740697 8963 raft_consensus.cc:2781] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:25.742484 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a365c229ed9c4c01987b086db659d5f8" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:25.742691 7484 raft_consensus.cc:3037] T a365c229ed9c4c01987b086db659d5f8 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:25.745792 7484 raft_consensus.cc:2454] T a365c229ed9c4c01987b086db659d5f8 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
I20240502 21:27:25.755204 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a365c229ed9c4c01987b086db659d5f8" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:25.755456 7483 raft_consensus.cc:2362] T a365c229ed9c4c01987b086db659d5f8 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d0456c5e55a542128688904e003d403c in term 1. Re-sending same reply.
I20240502 21:27:25.757126 9321 tablet_bootstrap.cc:492] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:25.740821 8963 raft_consensus.cc:686] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Becoming Leader. State: Replica: d0456c5e55a542128688904e003d403c, State: Running, Role: LEADER
I20240502 21:27:25.757766 8963 consensus_queue.cc:237] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:25.761560 9307 tablet_bootstrap.cc:492] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:25.763182 9307 tablet_bootstrap.cc:654] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:25.764732 7194 tablet_service.cc:1467] Processing CreateTablet for tablet 64aeda4ec20a4b568a475cb380d2568c (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=c758184ce285455e9f74bd2b6fd9dfc7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:25.765129 9321 tablet_bootstrap.cc:654] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:25.765302 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64aeda4ec20a4b568a475cb380d2568c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:25.766602 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a365c229ed9c4c01987b086db659d5f8" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:27:25.766862 4705 raft_consensus.cc:2379] T a365c229ed9c4c01987b086db659d5f8 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31a72286397c4edc9a63eca3412be168 in current term 1: Already voted for candidate d0456c5e55a542128688904e003d403c in this term.
I20240502 21:27:25.770907 7420 leader_election.cc:302] T a365c229ed9c4c01987b086db659d5f8 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a72286397c4edc9a63eca3412be168; no voters: 398c3c8f3195492c900439ad5e1931bb, d0456c5e55a542128688904e003d403c
I20240502 21:27:25.771459 9225 raft_consensus.cc:2726] T a365c229ed9c4c01987b086db659d5f8 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:27:25.772090 9321 tablet_bootstrap.cc:492] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:25.773176 9321 ts_tablet_manager.cc:1367] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.001s
I20240502 21:27:25.773813 9305 tablet_bootstrap.cc:492] T 64aeda4ec20a4b568a475cb380d2568c P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:25.775660 9321 raft_consensus.cc:348] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:25.776010 9305 tablet_bootstrap.cc:654] T 64aeda4ec20a4b568a475cb380d2568c P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:25.776655 9321 raft_consensus.cc:374] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:25.777098 9321 raft_consensus.cc:729] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:25.777951 9321 consensus_queue.cc:260] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:25.781023 9321 ts_tablet_manager.cc:1398] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 21:27:25.782174 9321 tablet_bootstrap.cc:492] T 64aeda4ec20a4b568a475cb380d2568c P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:25.787451 9321 tablet_bootstrap.cc:654] T 64aeda4ec20a4b568a475cb380d2568c P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:25.798843 9308 tablet_bootstrap.cc:492] T 180e178d0c714a35b8ca8843fad466c0 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:25.799082 9308 ts_tablet_manager.cc:1367] T 180e178d0c714a35b8ca8843fad466c0 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.064s user 0.011s sys 0.012s
I20240502 21:27:25.799304 9317 tablet_bootstrap.cc:492] T 180e178d0c714a35b8ca8843fad466c0 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:25.800038 9308 raft_consensus.cc:348] T 180e178d0c714a35b8ca8843fad466c0 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:25.800290 9308 raft_consensus.cc:374] T 180e178d0c714a35b8ca8843fad466c0 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:25.800372 9308 raft_consensus.cc:729] T 180e178d0c714a35b8ca8843fad466c0 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:25.800779 9308 consensus_queue.cc:260] T 180e178d0c714a35b8ca8843fad466c0 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:25.801510 9308 ts_tablet_manager.cc:1398] T 180e178d0c714a35b8ca8843fad466c0 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:25.802413 9307 tablet_bootstrap.cc:492] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:25.802609 9307 ts_tablet_manager.cc:1367] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.001s
I20240502 21:27:25.803776 9307 raft_consensus.cc:348] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:25.804078 9307 raft_consensus.cc:374] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:25.804214 9307 raft_consensus.cc:729] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:25.804606 9307 consensus_queue.cc:260] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:25.805174 9307 ts_tablet_manager.cc:1398] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:25.807337 9317 tablet_bootstrap.cc:654] T 180e178d0c714a35b8ca8843fad466c0 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:25.808292 9305 tablet_bootstrap.cc:492] T 64aeda4ec20a4b568a475cb380d2568c P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:27:25.808784 9305 ts_tablet_manager.cc:1367] T 64aeda4ec20a4b568a475cb380d2568c P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.000s
I20240502 21:27:25.809378 9321 tablet_bootstrap.cc:492] T 64aeda4ec20a4b568a475cb380d2568c P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:25.809540 9321 ts_tablet_manager.cc:1367] T 64aeda4ec20a4b568a475cb380d2568c P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.004s
I20240502 21:27:25.809669 9305 raft_consensus.cc:348] T 64aeda4ec20a4b568a475cb380d2568c P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:25.809885 9305 raft_consensus.cc:374] T 64aeda4ec20a4b568a475cb380d2568c P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:25.809967 9305 raft_consensus.cc:729] T 64aeda4ec20a4b568a475cb380d2568c P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:25.810052 8963 raft_consensus.cc:482] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:25.810194 8963 raft_consensus.cc:504] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:25.810196 9305 consensus_queue.cc:260] T 64aeda4ec20a4b568a475cb380d2568c P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:25.810812 9305 ts_tablet_manager.cc:1398] T 64aeda4ec20a4b568a475cb380d2568c P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:25.811448 7616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "180e178d0c714a35b8ca8843fad466c0" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:25.811681 8963 leader_election.cc:288] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
W20240502 21:27:25.812175 7282 leader_election.cc:341] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:25.812184 9321 raft_consensus.cc:348] T 64aeda4ec20a4b568a475cb380d2568c P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:25.812673 9321 raft_consensus.cc:374] T 64aeda4ec20a4b568a475cb380d2568c P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:25.812952 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "180e178d0c714a35b8ca8843fad466c0" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:25.813189 7483 raft_consensus.cc:2454] T 180e178d0c714a35b8ca8843fad466c0 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 0.
I20240502 21:27:25.814357 7281 leader_election.cc:302] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, d0456c5e55a542128688904e003d403c; no voters: 0ddd486109f5434cace13d60d633ac04
I20240502 21:27:25.815093 9198 raft_consensus.cc:2781] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:25.815193 9198 raft_consensus.cc:482] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:25.815290 9198 raft_consensus.cc:3037] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:25.816181 9316 tablet_bootstrap.cc:492] T 202f8b10d5674afa875da2f3929bd8da P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:25.816336 9316 ts_tablet_manager.cc:1367] T 202f8b10d5674afa875da2f3929bd8da P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.093s user 0.011s sys 0.007s
I20240502 21:27:25.817072 9316 raft_consensus.cc:348] T 202f8b10d5674afa875da2f3929bd8da P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:25.817373 9316 raft_consensus.cc:374] T 202f8b10d5674afa875da2f3929bd8da P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:25.817515 9316 raft_consensus.cc:729] T 202f8b10d5674afa875da2f3929bd8da P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:25.817833 9316 consensus_queue.cc:260] T 202f8b10d5674afa875da2f3929bd8da P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:25.818269 9198 raft_consensus.cc:504] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:25.818660 9316 ts_tablet_manager.cc:1398] T 202f8b10d5674afa875da2f3929bd8da P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:25.818631 4291 catalog_manager.cc:5484] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c reported cstate change: term changed from 0 to 1, leader changed from <none> to d0456c5e55a542128688904e003d403c (127.29.82.70). New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:25.818950 9316 tablet_bootstrap.cc:492] T 64aeda4ec20a4b568a475cb380d2568c P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:25.821827 9316 tablet_bootstrap.cc:654] T 64aeda4ec20a4b568a475cb380d2568c P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:25.823963 9198 leader_election.cc:288] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Requested vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:25.824776 7482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "180e178d0c714a35b8ca8843fad466c0" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:25.824776 7617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "180e178d0c714a35b8ca8843fad466c0" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:25.825011 7482 raft_consensus.cc:3037] T 180e178d0c714a35b8ca8843fad466c0 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:27:25.827816 7282 leader_election.cc:341] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:25.831341 7482 raft_consensus.cc:2454] T 180e178d0c714a35b8ca8843fad466c0 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
I20240502 21:27:25.812750 9321 raft_consensus.cc:729] T 64aeda4ec20a4b568a475cb380d2568c P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:25.831842 7281 leader_election.cc:302] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, d0456c5e55a542128688904e003d403c; no voters: 0ddd486109f5434cace13d60d633ac04
I20240502 21:27:25.832353 9321 consensus_queue.cc:260] T 64aeda4ec20a4b568a475cb380d2568c P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:25.832726 9198 raft_consensus.cc:2781] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:25.833127 9198 raft_consensus.cc:686] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Becoming Leader. State: Replica: d0456c5e55a542128688904e003d403c, State: Running, Role: LEADER
I20240502 21:27:25.833384 9198 consensus_queue.cc:237] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:25.852320 4291 catalog_manager.cc:5484] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c reported cstate change: term changed from 0 to 1, leader changed from <none> to d0456c5e55a542128688904e003d403c (127.29.82.70). New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:27:25.856009 9321 ts_tablet_manager.cc:1398] T 64aeda4ec20a4b568a475cb380d2568c P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.046s user 0.001s sys 0.001s
I20240502 21:27:25.859362 9246 raft_consensus.cc:482] T 64aeda4ec20a4b568a475cb380d2568c P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:25.859503 9246 raft_consensus.cc:504] T 64aeda4ec20a4b568a475cb380d2568c P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:25.860396 9246 leader_election.cc:288] T 64aeda4ec20a4b568a475cb380d2568c P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:25.861217 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64aeda4ec20a4b568a475cb380d2568c" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:27:25.861452 4705 raft_consensus.cc:2454] T 64aeda4ec20a4b568a475cb380d2568c P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:27:25.862111 4773 leader_election.cc:302] T 64aeda4ec20a4b568a475cb380d2568c P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:25.862740 9246 raft_consensus.cc:2781] T 64aeda4ec20a4b568a475cb380d2568c P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:25.862812 9246 raft_consensus.cc:482] T 64aeda4ec20a4b568a475cb380d2568c P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:25.862864 9246 raft_consensus.cc:3037] T 64aeda4ec20a4b568a475cb380d2568c P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:25.870088 9246 raft_consensus.cc:504] T 64aeda4ec20a4b568a475cb380d2568c P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:25.870342 7213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64aeda4ec20a4b568a475cb380d2568c" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:25.871706 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64aeda4ec20a4b568a475cb380d2568c" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:27:25.871908 4705 raft_consensus.cc:3037] T 64aeda4ec20a4b568a475cb380d2568c P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:27:25.874218 4772 leader_election.cc:341] T 64aeda4ec20a4b568a475cb380d2568c P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:25.874694 4705 raft_consensus.cc:2454] T 64aeda4ec20a4b568a475cb380d2568c P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:27:25.876775 4773 leader_election.cc:302] T 64aeda4ec20a4b568a475cb380d2568c P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:25.877036 8561 raft_consensus.cc:2781] T 64aeda4ec20a4b568a475cb380d2568c P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:25.877177 8561 raft_consensus.cc:686] T 64aeda4ec20a4b568a475cb380d2568c P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:27:25.877930 9317 tablet_bootstrap.cc:492] T 180e178d0c714a35b8ca8843fad466c0 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:25.878665 9317 ts_tablet_manager.cc:1367] T 180e178d0c714a35b8ca8843fad466c0 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.080s user 0.007s sys 0.000s
I20240502 21:27:25.879453 9316 tablet_bootstrap.cc:492] T 64aeda4ec20a4b568a475cb380d2568c P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:25.879545 8561 consensus_queue.cc:237] T 64aeda4ec20a4b568a475cb380d2568c P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:25.879864 9316 ts_tablet_manager.cc:1367] T 64aeda4ec20a4b568a475cb380d2568c P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.061s user 0.005s sys 0.007s
I20240502 21:27:25.881311 9316 raft_consensus.cc:348] T 64aeda4ec20a4b568a475cb380d2568c P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:25.881412 9317 raft_consensus.cc:348] T 180e178d0c714a35b8ca8843fad466c0 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:25.881661 9316 raft_consensus.cc:374] T 64aeda4ec20a4b568a475cb380d2568c P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:25.881736 9317 raft_consensus.cc:374] T 180e178d0c714a35b8ca8843fad466c0 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:25.881788 9316 raft_consensus.cc:729] T 64aeda4ec20a4b568a475cb380d2568c P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:25.881857 9317 raft_consensus.cc:729] T 180e178d0c714a35b8ca8843fad466c0 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:25.882090 9316 consensus_queue.cc:260] T 64aeda4ec20a4b568a475cb380d2568c P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:25.882107 9317 consensus_queue.cc:260] T 180e178d0c714a35b8ca8843fad466c0 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:25.882781 9316 ts_tablet_manager.cc:1398] T 64aeda4ec20a4b568a475cb380d2568c P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:25.884069 9317 ts_tablet_manager.cc:1398] T 180e178d0c714a35b8ca8843fad466c0 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 21:27:25.884400 9317 tablet_bootstrap.cc:492] T 202f8b10d5674afa875da2f3929bd8da P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:25.886029 9317 tablet_bootstrap.cc:654] T 202f8b10d5674afa875da2f3929bd8da P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:25.900111 4294 catalog_manager.cc:5484] T 64aeda4ec20a4b568a475cb380d2568c P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:25.910934 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64aeda4ec20a4b568a475cb380d2568c" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:25.911195 7214 raft_consensus.cc:3037] T 64aeda4ec20a4b568a475cb380d2568c P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:25.914017 7214 raft_consensus.cc:2454] T 64aeda4ec20a4b568a475cb380d2568c P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:27:25.916572 9246 leader_election.cc:288] T 64aeda4ec20a4b568a475cb380d2568c P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:25.927417 9317 tablet_bootstrap.cc:492] T 202f8b10d5674afa875da2f3929bd8da P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:25.927814 9317 ts_tablet_manager.cc:1367] T 202f8b10d5674afa875da2f3929bd8da P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.043s user 0.020s sys 0.016s
I20240502 21:27:25.928819 9317 raft_consensus.cc:348] T 202f8b10d5674afa875da2f3929bd8da P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:25.929100 9317 raft_consensus.cc:374] T 202f8b10d5674afa875da2f3929bd8da P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:25.929210 9317 raft_consensus.cc:729] T 202f8b10d5674afa875da2f3929bd8da P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:25.929515 9317 consensus_queue.cc:260] T 202f8b10d5674afa875da2f3929bd8da P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:25.930213 9317 ts_tablet_manager.cc:1398] T 202f8b10d5674afa875da2f3929bd8da P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:25.931593 9150 raft_consensus.cc:482] T 180e178d0c714a35b8ca8843fad466c0 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:25.931730 9150 raft_consensus.cc:504] T 180e178d0c714a35b8ca8843fad466c0 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:25.932451 9150 leader_election.cc:288] T 180e178d0c714a35b8ca8843fad466c0 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:25.933142 7348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "180e178d0c714a35b8ca8843fad466c0" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:25.934500 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "180e178d0c714a35b8ca8843fad466c0" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:25.934767 7483 raft_consensus.cc:2379] T 180e178d0c714a35b8ca8843fad466c0 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ddd486109f5434cace13d60d633ac04 in current term 1: Already voted for candidate d0456c5e55a542128688904e003d403c in this term.
I20240502 21:27:25.935218 7549 leader_election.cc:302] T 180e178d0c714a35b8ca8843fad466c0 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04; no voters: 31a72286397c4edc9a63eca3412be168, d0456c5e55a542128688904e003d403c
I20240502 21:27:25.935726 9071 raft_consensus.cc:3037] T 180e178d0c714a35b8ca8843fad466c0 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:25.938745 9071 raft_consensus.cc:2726] T 180e178d0c714a35b8ca8843fad466c0 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:27:25.970912 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:25.977389 4705 raft_consensus.cc:1261] T a365c229ed9c4c01987b086db659d5f8 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:25.977458 7483 raft_consensus.cc:1261] T a365c229ed9c4c01987b086db659d5f8 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:25.977933 8963 consensus_queue.cc:1035] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:25.978680 8963 consensus_queue.cc:1035] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:25.986248 9339 tablet.cc:1722] T a365c229ed9c4c01987b086db659d5f8 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:25.987320 9339 tablet.cc:1620] T a365c229ed9c4c01987b086db659d5f8 P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:25.990028 4294 catalog_manager.cc:5576] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913) does not have the latest schema for tablet a365c229ed9c4c01987b086db659d5f8 (table rebalancer_dynamic_table [id=24623a600ef24bddb541ee187c709932]). Expected version 1 got 0
I20240502 21:27:25.992269 9338 tablet.cc:1722] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:25.992856 9338 tablet.cc:1620] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:26.001725 4294 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=24623a600ef24bddb541ee187c709932] alter complete (version 1)
I20240502 21:27:26.021050 4294 catalog_manager.cc:5576] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537) does not have the latest schema for tablet a365c229ed9c4c01987b086db659d5f8 (table rebalancer_dynamic_table [id=24623a600ef24bddb541ee187c709932]). Expected version 1 got 0
I20240502 21:27:26.022388 9340 tablet.cc:1722] T a365c229ed9c4c01987b086db659d5f8 P 398c3c8f3195492c900439ad5e1931bb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:26.023226 9340 tablet.cc:1620] T a365c229ed9c4c01987b086db659d5f8 P 398c3c8f3195492c900439ad5e1931bb: MemRowSet was empty: no flush needed.
I20240502 21:27:26.103204 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:26.110700 9338 tablet.cc:1722] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:26.111459 9338 tablet.cc:1620] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c: MemRowSet was empty: no flush needed.
I20240502 21:27:26.111727 9340 tablet.cc:1722] T a365c229ed9c4c01987b086db659d5f8 P 398c3c8f3195492c900439ad5e1931bb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:26.112288 9340 tablet.cc:1620] T a365c229ed9c4c01987b086db659d5f8 P 398c3c8f3195492c900439ad5e1931bb: MemRowSet was empty: no flush needed.
I20240502 21:27:26.112608 9339 tablet.cc:1722] T a365c229ed9c4c01987b086db659d5f8 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:26.113353 9339 tablet.cc:1620] T a365c229ed9c4c01987b086db659d5f8 P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:26.122334 4291 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=24623a600ef24bddb541ee187c709932] alter complete (version 2)
I20240502 21:27:26.160849 9246 raft_consensus.cc:482] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:26.161006 9246 raft_consensus.cc:504] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:26.161625 9246 leader_election.cc:288] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:26.162043 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "202f8b10d5674afa875da2f3929bd8da" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:26.162132 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "202f8b10d5674afa875da2f3929bd8da" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:26.162328 7214 raft_consensus.cc:2454] T 202f8b10d5674afa875da2f3929bd8da P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:27:26.162353 7618 raft_consensus.cc:2454] T 202f8b10d5674afa875da2f3929bd8da P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 0.
I20240502 21:27:26.162715 4772 leader_election.cc:302] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 39981a8dfe94462b9a73ba489af4c714, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:26.162971 9246 raft_consensus.cc:2781] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:26.163066 9246 raft_consensus.cc:482] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:26.163142 9246 raft_consensus.cc:3037] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:26.165846 9246 raft_consensus.cc:504] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:26.166461 9246 leader_election.cc:288] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Requested vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:26.166813 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "202f8b10d5674afa875da2f3929bd8da" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:26.166962 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "202f8b10d5674afa875da2f3929bd8da" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:26.167040 7214 raft_consensus.cc:3037] T 202f8b10d5674afa875da2f3929bd8da P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:26.167186 7618 raft_consensus.cc:3037] T 202f8b10d5674afa875da2f3929bd8da P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:26.169919 7214 raft_consensus.cc:2454] T 202f8b10d5674afa875da2f3929bd8da P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:27:26.170022 7618 raft_consensus.cc:2454] T 202f8b10d5674afa875da2f3929bd8da P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:27:26.170295 4772 leader_election.cc:302] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:26.170575 9246 raft_consensus.cc:2781] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:26.170745 9246 raft_consensus.cc:686] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:27:26.170984 9246 consensus_queue.cc:237] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:26.176177 4294 catalog_manager.cc:5484] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:26.222362 8963 consensus_queue.cc:1035] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:26.228006 4291 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:26.230276 8963 consensus_queue.cc:1035] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:26.240993 9338 tablet.cc:1722] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:26.243301 9340 tablet.cc:1722] T a365c229ed9c4c01987b086db659d5f8 P 398c3c8f3195492c900439ad5e1931bb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:26.246217 9339 tablet.cc:1722] T a365c229ed9c4c01987b086db659d5f8 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:26.254848 4292 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=24623a600ef24bddb541ee187c709932] alter complete (version 3)
I20240502 21:27:26.361874 4291 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:26.362130 4291 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:26.365177 4291 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet a365c229ed9c4c01987b086db659d5f8
I20240502 21:27:26.366325 7328 tablet_service.cc:1514] Processing DeleteTablet for tablet a365c229ed9c4c01987b086db659d5f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:26 UTC) from {username='slave'} at 127.0.0.1:47078
I20240502 21:27:26.366338 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet a365c229ed9c4c01987b086db659d5f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:26 UTC) from {username='slave'} at 127.0.0.1:44764
I20240502 21:27:26.366343 7462 tablet_service.cc:1514] Processing DeleteTablet for tablet a365c229ed9c4c01987b086db659d5f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:26 UTC) from {username='slave'} at 127.0.0.1:37182
I20240502 21:27:26.382328 9346 tablet_replica.cc:330] T a365c229ed9c4c01987b086db659d5f8 P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:27:26.382740 9345 tablet_replica.cc:330] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:27:26.382791 9347 tablet_replica.cc:330] T a365c229ed9c4c01987b086db659d5f8 P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:27:26.382972 9345 raft_consensus.cc:2229] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:26.383107 9347 raft_consensus.cc:2229] T a365c229ed9c4c01987b086db659d5f8 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:26.383322 9347 raft_consensus.cc:2258] T a365c229ed9c4c01987b086db659d5f8 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:26.383374 9345 raft_consensus.cc:2258] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:26.383706 9346 raft_consensus.cc:2229] T a365c229ed9c4c01987b086db659d5f8 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:26.383955 9346 raft_consensus.cc:2258] T a365c229ed9c4c01987b086db659d5f8 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:26.384454 9345 ts_tablet_manager.cc:1875] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:26.384867 9347 ts_tablet_manager.cc:1875] T a365c229ed9c4c01987b086db659d5f8 P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:26.384956 9346 ts_tablet_manager.cc:1875] T a365c229ed9c4c01987b086db659d5f8 P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:26.391470 9346 ts_tablet_manager.cc:1888] T a365c229ed9c4c01987b086db659d5f8 P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:26.391728 9346 log.cc:1196] T a365c229ed9c4c01987b086db659d5f8 P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/a365c229ed9c4c01987b086db659d5f8
I20240502 21:27:26.392319 9346 ts_tablet_manager.cc:1909] T a365c229ed9c4c01987b086db659d5f8 P 398c3c8f3195492c900439ad5e1931bb: Deleting consensus metadata
I20240502 21:27:26.392802 9347 ts_tablet_manager.cc:1888] T a365c229ed9c4c01987b086db659d5f8 P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:26.393000 9347 log.cc:1196] T a365c229ed9c4c01987b086db659d5f8 P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/a365c229ed9c4c01987b086db659d5f8
I20240502 21:27:26.393541 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet a365c229ed9c4c01987b086db659d5f8 (table rebalancer_dynamic_table_new_name [id=24623a600ef24bddb541ee187c709932]) successfully deleted
I20240502 21:27:26.394053 9347 ts_tablet_manager.cc:1909] T a365c229ed9c4c01987b086db659d5f8 P 31a72286397c4edc9a63eca3412be168: Deleting consensus metadata
I20240502 21:27:26.396313 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet a365c229ed9c4c01987b086db659d5f8 (table rebalancer_dynamic_table_new_name [id=24623a600ef24bddb541ee187c709932]) successfully deleted
I20240502 21:27:26.401571 9345 ts_tablet_manager.cc:1888] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:26.401693 9345 log.cc:1196] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/a365c229ed9c4c01987b086db659d5f8
I20240502 21:27:26.402259 9345 ts_tablet_manager.cc:1909] T a365c229ed9c4c01987b086db659d5f8 P d0456c5e55a542128688904e003d403c: Deleting consensus metadata
I20240502 21:27:26.404620 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet a365c229ed9c4c01987b086db659d5f8 (table rebalancer_dynamic_table_new_name [id=24623a600ef24bddb541ee187c709932]) successfully deleted
I20240502 21:27:26.422215 9246 consensus_queue.cc:1035] T 64aeda4ec20a4b568a475cb380d2568c P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:26.429440 8561 consensus_queue.cc:1035] T 64aeda4ec20a4b568a475cb380d2568c P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:26.474684 4298 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:26.500083 7597 tablet_service.cc:1467] Processing CreateTablet for tablet 7c2c819c0a274a80b309531af6733885 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6da95202b2c94de5b120f347345ed051]), partition=
I20240502 21:27:26.500458 7462 tablet_service.cc:1467] Processing CreateTablet for tablet 7c2c819c0a274a80b309531af6733885 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6da95202b2c94de5b120f347345ed051]), partition=
I20240502 21:27:26.500576 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c2c819c0a274a80b309531af6733885. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.500861 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c2c819c0a274a80b309531af6733885. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.501821 4555 tablet_service.cc:1467] Processing CreateTablet for tablet 7c2c819c0a274a80b309531af6733885 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6da95202b2c94de5b120f347345ed051]), partition=
I20240502 21:27:26.502200 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c2c819c0a274a80b309531af6733885. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.515836 9356 tablet_bootstrap.cc:492] T 7c2c819c0a274a80b309531af6733885 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:26.518663 9354 tablet_bootstrap.cc:492] T 7c2c819c0a274a80b309531af6733885 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:26.519392 9356 tablet_bootstrap.cc:654] T 7c2c819c0a274a80b309531af6733885 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:26.520398 9354 tablet_bootstrap.cc:654] T 7c2c819c0a274a80b309531af6733885 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:26.521389 9356 tablet_bootstrap.cc:492] T 7c2c819c0a274a80b309531af6733885 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:26.521538 9356 ts_tablet_manager.cc:1367] T 7c2c819c0a274a80b309531af6733885 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 21:27:26.521843 9355 tablet_bootstrap.cc:492] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:26.523234 9356 raft_consensus.cc:348] T 7c2c819c0a274a80b309531af6733885 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:26.523550 9356 raft_consensus.cc:374] T 7c2c819c0a274a80b309531af6733885 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:26.523696 9356 raft_consensus.cc:729] T 7c2c819c0a274a80b309531af6733885 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:26.523850 9355 tablet_bootstrap.cc:654] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:26.523967 9356 consensus_queue.cc:260] T 7c2c819c0a274a80b309531af6733885 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:26.524698 9356 ts_tablet_manager.cc:1398] T 7c2c819c0a274a80b309531af6733885 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:26.526342 9354 tablet_bootstrap.cc:492] T 7c2c819c0a274a80b309531af6733885 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:26.526569 9354 ts_tablet_manager.cc:1367] T 7c2c819c0a274a80b309531af6733885 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240502 21:27:26.527680 9354 raft_consensus.cc:348] T 7c2c819c0a274a80b309531af6733885 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:26.527962 9354 raft_consensus.cc:374] T 7c2c819c0a274a80b309531af6733885 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:26.528112 9354 raft_consensus.cc:729] T 7c2c819c0a274a80b309531af6733885 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:26.528486 9354 consensus_queue.cc:260] T 7c2c819c0a274a80b309531af6733885 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:26.529450 9355 tablet_bootstrap.cc:492] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:26.529670 9355 ts_tablet_manager.cc:1367] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20240502 21:27:26.530484 9355 raft_consensus.cc:348] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:26.530758 9355 raft_consensus.cc:374] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:26.530880 9355 raft_consensus.cc:729] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:26.531165 9355 consensus_queue.cc:260] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:26.531863 9355 ts_tablet_manager.cc:1398] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:26.533239 9354 ts_tablet_manager.cc:1398] T 7c2c819c0a274a80b309531af6733885 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 21:27:26.588747 8561 consensus_queue.cc:1035] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:26.593683 9246 consensus_queue.cc:1035] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:26.685632 4298 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:26.695183 7328 tablet_service.cc:1467] Processing CreateTablet for tablet 37c2c46368144212a0041e7ff4434236 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=2f504526d1174ad1ad3808fc9c728ce7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:26.695343 4422 tablet_service.cc:1467] Processing CreateTablet for tablet 37c2c46368144212a0041e7ff4434236 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=2f504526d1174ad1ad3808fc9c728ce7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:26.695736 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c2c46368144212a0041e7ff4434236. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.695833 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c2c46368144212a0041e7ff4434236. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.696570 7194 tablet_service.cc:1467] Processing CreateTablet for tablet d81b71825ba54eacb6c106bce3a838e8 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=2f504526d1174ad1ad3808fc9c728ce7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:26.696983 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81b71825ba54eacb6c106bce3a838e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.699963 7195 tablet_service.cc:1467] Processing CreateTablet for tablet 0914e3ce5a2d43788ba9d369227e4f66 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=2f504526d1174ad1ad3808fc9c728ce7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:26.700016 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 37c2c46368144212a0041e7ff4434236 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=2f504526d1174ad1ad3808fc9c728ce7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:26.700042 4555 tablet_service.cc:1467] Processing CreateTablet for tablet d81b71825ba54eacb6c106bce3a838e8 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=2f504526d1174ad1ad3808fc9c728ce7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:26.700357 7195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0914e3ce5a2d43788ba9d369227e4f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.700443 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c2c46368144212a0041e7ff4434236. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.700543 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81b71825ba54eacb6c106bce3a838e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.706779 9225 raft_consensus.cc:482] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:26.706954 9225 raft_consensus.cc:504] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:26.707718 9225 leader_election.cc:288] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:27:26.709126 9356 tablet_bootstrap.cc:492] T d81b71825ba54eacb6c106bce3a838e8 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:26.709062 7616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2c819c0a274a80b309531af6733885" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:27:26.709285 7616 raft_consensus.cc:2454] T 7c2c819c0a274a80b309531af6733885 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:26.709868 7418 leader_election.cc:302] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168; no voters:
I20240502 21:27:26.710247 9225 raft_consensus.cc:2781] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:26.710363 9225 raft_consensus.cc:482] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:26.710502 9225 raft_consensus.cc:3037] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:26.710996 9356 tablet_bootstrap.cc:654] T d81b71825ba54eacb6c106bce3a838e8 P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:26.712245 9363 tablet_bootstrap.cc:492] T 37c2c46368144212a0041e7ff4434236 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:26.713291 9356 tablet_bootstrap.cc:492] T d81b71825ba54eacb6c106bce3a838e8 P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:26.713517 9356 ts_tablet_manager.cc:1367] T d81b71825ba54eacb6c106bce3a838e8 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20240502 21:27:26.713946 9363 tablet_bootstrap.cc:654] T 37c2c46368144212a0041e7ff4434236 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:26.714089 9225 raft_consensus.cc:504] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:26.714534 9356 raft_consensus.cc:348] T d81b71825ba54eacb6c106bce3a838e8 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:26.714823 9225 leader_election.cc:288] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Requested vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:27:26.714974 9356 raft_consensus.cc:374] T d81b71825ba54eacb6c106bce3a838e8 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:26.715133 9356 raft_consensus.cc:729] T d81b71825ba54eacb6c106bce3a838e8 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:26.715369 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2c819c0a274a80b309531af6733885" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:27:26.715634 4575 raft_consensus.cc:2454] T 7c2c819c0a274a80b309531af6733885 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:26.716688 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2c819c0a274a80b309531af6733885" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:27:26.717424 4575 raft_consensus.cc:3037] T 7c2c819c0a274a80b309531af6733885 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:26.720005 9356 consensus_queue.cc:260] T d81b71825ba54eacb6c106bce3a838e8 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:26.720044 7616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2c819c0a274a80b309531af6733885" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:27:26.720366 7616 raft_consensus.cc:3037] T 7c2c819c0a274a80b309531af6733885 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:26.728602 9356 ts_tablet_manager.cc:1398] T d81b71825ba54eacb6c106bce3a838e8 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20240502 21:27:26.738279 9362 tablet_bootstrap.cc:492] T 37c2c46368144212a0041e7ff4434236 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:26.741458 4422 tablet_service.cc:1467] Processing CreateTablet for tablet 0914e3ce5a2d43788ba9d369227e4f66 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=2f504526d1174ad1ad3808fc9c728ce7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:26.741524 7462 tablet_service.cc:1467] Processing CreateTablet for tablet d81b71825ba54eacb6c106bce3a838e8 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=2f504526d1174ad1ad3808fc9c728ce7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:26.741978 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81b71825ba54eacb6c106bce3a838e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.742005 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0914e3ce5a2d43788ba9d369227e4f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.745137 4817 tablet_service.cc:1467] Processing CreateTablet for tablet 0914e3ce5a2d43788ba9d369227e4f66 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=2f504526d1174ad1ad3808fc9c728ce7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:26.745671 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0914e3ce5a2d43788ba9d369227e4f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.763957 9362 tablet_bootstrap.cc:654] T 37c2c46368144212a0041e7ff4434236 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:26.769599 9363 tablet_bootstrap.cc:492] T 37c2c46368144212a0041e7ff4434236 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:26.769842 9363 ts_tablet_manager.cc:1367] T 37c2c46368144212a0041e7ff4434236 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.058s user 0.013s sys 0.021s
I20240502 21:27:26.770746 9363 raft_consensus.cc:348] T 37c2c46368144212a0041e7ff4434236 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:26.771020 9363 raft_consensus.cc:374] T 37c2c46368144212a0041e7ff4434236 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:26.771072 9363 raft_consensus.cc:729] T 37c2c46368144212a0041e7ff4434236 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:26.771399 9363 consensus_queue.cc:260] T 37c2c46368144212a0041e7ff4434236 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:26.772063 4575 raft_consensus.cc:2454] T 7c2c819c0a274a80b309531af6733885 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:26.772131 9363 ts_tablet_manager.cc:1398] T 37c2c46368144212a0041e7ff4434236 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:26.772554 7420 leader_election.cc:302] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:27:26.772929 9225 raft_consensus.cc:2781] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:26.773072 9225 raft_consensus.cc:686] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Becoming Leader. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Running, Role: LEADER
I20240502 21:27:26.773381 9225 consensus_queue.cc:237] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:26.774960 9363 tablet_bootstrap.cc:492] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:26.777094 9363 tablet_bootstrap.cc:654] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:26.772301 9370 tablet_bootstrap.cc:492] T 0914e3ce5a2d43788ba9d369227e4f66 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:26.779693 9362 tablet_bootstrap.cc:492] T 37c2c46368144212a0041e7ff4434236 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:26.779817 9370 tablet_bootstrap.cc:654] T 0914e3ce5a2d43788ba9d369227e4f66 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:26.779983 9362 ts_tablet_manager.cc:1367] T 37c2c46368144212a0041e7ff4434236 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.009s
I20240502 21:27:26.780979 9362 raft_consensus.cc:348] T 37c2c46368144212a0041e7ff4434236 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:26.781368 9362 raft_consensus.cc:374] T 37c2c46368144212a0041e7ff4434236 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:26.781531 9362 raft_consensus.cc:729] T 37c2c46368144212a0041e7ff4434236 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:26.781908 9362 consensus_queue.cc:260] T 37c2c46368144212a0041e7ff4434236 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:26.782685 9362 ts_tablet_manager.cc:1398] T 37c2c46368144212a0041e7ff4434236 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:26.783308 7616 raft_consensus.cc:2454] T 7c2c819c0a274a80b309531af6733885 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:26.784416 9355 tablet_bootstrap.cc:492] T d81b71825ba54eacb6c106bce3a838e8 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:26.786154 9355 tablet_bootstrap.cc:654] T d81b71825ba54eacb6c106bce3a838e8 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:26.786904 9370 tablet_bootstrap.cc:492] T 0914e3ce5a2d43788ba9d369227e4f66 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:26.787109 9370 ts_tablet_manager.cc:1367] T 0914e3ce5a2d43788ba9d369227e4f66 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240502 21:27:26.789273 9363 tablet_bootstrap.cc:492] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:26.790037 9370 raft_consensus.cc:348] T 0914e3ce5a2d43788ba9d369227e4f66 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:26.790414 9363 ts_tablet_manager.cc:1367] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240502 21:27:26.790360 4294 catalog_manager.cc:5484] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a72286397c4edc9a63eca3412be168 (127.29.82.71). New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:26.791213 9363 raft_consensus.cc:348] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:26.791460 9370 raft_consensus.cc:374] T 0914e3ce5a2d43788ba9d369227e4f66 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:26.791600 9370 raft_consensus.cc:729] T 0914e3ce5a2d43788ba9d369227e4f66 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:26.791868 9370 consensus_queue.cc:260] T 0914e3ce5a2d43788ba9d369227e4f66 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:26.792121 9363 raft_consensus.cc:374] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:26.792238 9363 raft_consensus.cc:729] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:26.793582 9355 tablet_bootstrap.cc:492] T d81b71825ba54eacb6c106bce3a838e8 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:26.793901 9363 consensus_queue.cc:260] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:26.794401 9370 ts_tablet_manager.cc:1398] T 0914e3ce5a2d43788ba9d369227e4f66 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:27:26.794342 9355 ts_tablet_manager.cc:1367] T d81b71825ba54eacb6c106bce3a838e8 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240502 21:27:26.795493 9363 ts_tablet_manager.cc:1398] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:27:26.799383 9355 raft_consensus.cc:348] T d81b71825ba54eacb6c106bce3a838e8 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:26.803155 9355 raft_consensus.cc:374] T d81b71825ba54eacb6c106bce3a838e8 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:26.803457 9355 raft_consensus.cc:729] T d81b71825ba54eacb6c106bce3a838e8 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:26.803985 9355 consensus_queue.cc:260] T d81b71825ba54eacb6c106bce3a838e8 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:26.805096 9355 ts_tablet_manager.cc:1398] T d81b71825ba54eacb6c106bce3a838e8 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 21:27:26.810051 9364 tablet_bootstrap.cc:492] T 37c2c46368144212a0041e7ff4434236 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:26.810226 9369 tablet_bootstrap.cc:492] T d81b71825ba54eacb6c106bce3a838e8 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:26.812301 9364 tablet_bootstrap.cc:654] T 37c2c46368144212a0041e7ff4434236 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:26.814091 9369 tablet_bootstrap.cc:654] T d81b71825ba54eacb6c106bce3a838e8 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:26.820566 9369 tablet_bootstrap.cc:492] T d81b71825ba54eacb6c106bce3a838e8 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:26.820959 9369 ts_tablet_manager.cc:1367] T d81b71825ba54eacb6c106bce3a838e8 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240502 21:27:26.821905 9369 raft_consensus.cc:348] T d81b71825ba54eacb6c106bce3a838e8 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:26.822206 9369 raft_consensus.cc:374] T d81b71825ba54eacb6c106bce3a838e8 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:26.822383 9369 raft_consensus.cc:729] T d81b71825ba54eacb6c106bce3a838e8 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:26.825239 9369 consensus_queue.cc:260] T d81b71825ba54eacb6c106bce3a838e8 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:26.826117 9369 ts_tablet_manager.cc:1398] T d81b71825ba54eacb6c106bce3a838e8 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:27:26.830219 9369 tablet_bootstrap.cc:492] T 0914e3ce5a2d43788ba9d369227e4f66 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:26.832298 9369 tablet_bootstrap.cc:654] T 0914e3ce5a2d43788ba9d369227e4f66 P 39981a8dfe94462b9a73ba489af4c714: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:26.845770 9364 tablet_bootstrap.cc:492] T 37c2c46368144212a0041e7ff4434236 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:27:26.846025 9364 ts_tablet_manager.cc:1367] T 37c2c46368144212a0041e7ff4434236 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.000s
I20240502 21:27:26.848251 9364 raft_consensus.cc:348] T 37c2c46368144212a0041e7ff4434236 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:26.848568 9364 raft_consensus.cc:374] T 37c2c46368144212a0041e7ff4434236 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:26.848711 9364 raft_consensus.cc:729] T 37c2c46368144212a0041e7ff4434236 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:26.849042 9364 consensus_queue.cc:260] T 37c2c46368144212a0041e7ff4434236 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:26.853577 9364 ts_tablet_manager.cc:1398] T 37c2c46368144212a0041e7ff4434236 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 21:27:26.858135 9353 raft_consensus.cc:482] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:26.858309 9353 raft_consensus.cc:504] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:26.859153 9353 leader_election.cc:288] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:26.861294 4837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0914e3ce5a2d43788ba9d369227e4f66" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:27:26.861478 4837 raft_consensus.cc:2454] T 0914e3ce5a2d43788ba9d369227e4f66 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:26.863504 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0914e3ce5a2d43788ba9d369227e4f66" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:26.864105 4375 leader_election.cc:302] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
W20240502 21:27:26.865360 4377 leader_election.cc:341] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:26.865976 9353 raft_consensus.cc:2781] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:26.866073 9353 raft_consensus.cc:482] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:26.866154 9353 raft_consensus.cc:3037] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:26.868767 9369 tablet_bootstrap.cc:492] T 0914e3ce5a2d43788ba9d369227e4f66 P 39981a8dfe94462b9a73ba489af4c714: No bootstrap required, opened a new log
I20240502 21:27:26.868942 9369 ts_tablet_manager.cc:1367] T 0914e3ce5a2d43788ba9d369227e4f66 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.039s user 0.012s sys 0.009s
I20240502 21:27:26.869778 9369 raft_consensus.cc:348] T 0914e3ce5a2d43788ba9d369227e4f66 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:26.870015 9369 raft_consensus.cc:374] T 0914e3ce5a2d43788ba9d369227e4f66 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:26.870102 9369 raft_consensus.cc:729] T 0914e3ce5a2d43788ba9d369227e4f66 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:26.870365 9369 consensus_queue.cc:260] T 0914e3ce5a2d43788ba9d369227e4f66 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:26.870955 9369 ts_tablet_manager.cc:1398] T 0914e3ce5a2d43788ba9d369227e4f66 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:26.872866 9353 raft_consensus.cc:504] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:26.873840 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0914e3ce5a2d43788ba9d369227e4f66" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:27:26.874030 4839 raft_consensus.cc:3037] T 0914e3ce5a2d43788ba9d369227e4f66 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:26.876492 4839 raft_consensus.cc:2454] T 0914e3ce5a2d43788ba9d369227e4f66 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:26.879191 4375 leader_election.cc:302] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:26.879760 9036 raft_consensus.cc:2781] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:26.881287 9036 raft_consensus.cc:686] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:27:26.881610 9036 consensus_queue.cc:237] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:26.883859 9353 leader_election.cc:288] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:26.884752 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0914e3ce5a2d43788ba9d369227e4f66" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:26.885066 7214 raft_consensus.cc:3037] T 0914e3ce5a2d43788ba9d369227e4f66 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:26.888482 7214 raft_consensus.cc:2454] T 0914e3ce5a2d43788ba9d369227e4f66 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:26.891034 4294 catalog_manager.cc:5484] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:26.900148 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:26.903116 9314 raft_consensus.cc:482] T 37c2c46368144212a0041e7ff4434236 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:26.903324 9314 raft_consensus.cc:504] T 37c2c46368144212a0041e7ff4434236 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:26.904330 9314 leader_election.cc:288] T 37c2c46368144212a0041e7ff4434236 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:26.905905 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c2c46368144212a0041e7ff4434236" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:27:26.906136 4442 raft_consensus.cc:2454] T 37c2c46368144212a0041e7ff4434236 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:27:26.907809 4639 leader_election.cc:302] T 37c2c46368144212a0041e7ff4434236 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:26.910022 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c2c46368144212a0041e7ff4434236" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:26.910249 7349 raft_consensus.cc:2454] T 37c2c46368144212a0041e7ff4434236 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 0.
I20240502 21:27:26.911778 9314 raft_consensus.cc:2781] T 37c2c46368144212a0041e7ff4434236 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:26.911922 9314 raft_consensus.cc:482] T 37c2c46368144212a0041e7ff4434236 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:26.912021 9314 raft_consensus.cc:3037] T 37c2c46368144212a0041e7ff4434236 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:26.914424 9314 raft_consensus.cc:504] T 37c2c46368144212a0041e7ff4434236 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:26.915678 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c2c46368144212a0041e7ff4434236" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:27:26.915863 4442 raft_consensus.cc:3037] T 37c2c46368144212a0041e7ff4434236 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:26.918310 4442 raft_consensus.cc:2454] T 37c2c46368144212a0041e7ff4434236 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:27:26.918965 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c2c46368144212a0041e7ff4434236" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:26.919168 7349 raft_consensus.cc:3037] T 37c2c46368144212a0041e7ff4434236 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:26.921526 7349 raft_consensus.cc:2454] T 37c2c46368144212a0041e7ff4434236 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 1.
I20240502 21:27:26.921958 4641 leader_election.cc:302] T 37c2c46368144212a0041e7ff4434236 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:26.922298 9374 raft_consensus.cc:2781] T 37c2c46368144212a0041e7ff4434236 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:26.922456 9374 raft_consensus.cc:686] T 37c2c46368144212a0041e7ff4434236 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Becoming Leader. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Running, Role: LEADER
I20240502 21:27:26.922803 9374 consensus_queue.cc:237] T 37c2c46368144212a0041e7ff4434236 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:26.923763 9314 leader_election.cc:288] T 37c2c46368144212a0041e7ff4434236 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 1 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:26.927742 4291 catalog_manager.cc:5484] T 37c2c46368144212a0041e7ff4434236 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 398c3c8f3195492c900439ad5e1931bb (127.29.82.67). New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:26.928620 4575 raft_consensus.cc:1261] T 7c2c819c0a274a80b309531af6733885 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:26.930030 9367 consensus_queue.cc:1035] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:26.931929 7618 raft_consensus.cc:1261] T 7c2c819c0a274a80b309531af6733885 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:26.932544 9367 consensus_queue.cc:1035] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:26.938905 4294 catalog_manager.cc:5576] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781) does not have the latest schema for tablet 7c2c819c0a274a80b309531af6733885 (table rebalancer_dynamic_table [id=6da95202b2c94de5b120f347345ed051]). Expected version 1 got 0
I20240502 21:27:26.944051 9387 tablet.cc:1722] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:26.944734 9387 tablet.cc:1620] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:26.946415 4291 catalog_manager.cc:5576] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223) does not have the latest schema for tablet 7c2c819c0a274a80b309531af6733885 (table rebalancer_dynamic_table [id=6da95202b2c94de5b120f347345ed051]). Expected version 1 got 0
I20240502 21:27:26.953548 9388 tablet.cc:1722] T 7c2c819c0a274a80b309531af6733885 P 7a736a8ebb164ce0a051c0fccd76c3b8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:26.954185 9388 tablet.cc:1620] T 7c2c819c0a274a80b309531af6733885 P 7a736a8ebb164ce0a051c0fccd76c3b8: MemRowSet was empty: no flush needed.
I20240502 21:27:26.962366 4294 catalog_manager.cc:5576] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781) does not have the latest schema for tablet 7c2c819c0a274a80b309531af6733885 (table rebalancer_dynamic_table [id=6da95202b2c94de5b120f347345ed051]). Expected version 1 got 0
I20240502 21:27:26.963472 4291 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=6da95202b2c94de5b120f347345ed051] alter complete (version 1)
I20240502 21:27:26.967722 9390 tablet.cc:1722] T 7c2c819c0a274a80b309531af6733885 P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:26.968518 9390 tablet.cc:1620] T 7c2c819c0a274a80b309531af6733885 P 0ddd486109f5434cace13d60d633ac04: MemRowSet was empty: no flush needed.
I20240502 21:27:26.972571 4292 catalog_manager.cc:5576] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223) does not have the latest schema for tablet 7c2c819c0a274a80b309531af6733885 (table rebalancer_dynamic_table [id=6da95202b2c94de5b120f347345ed051]). Expected version 1 got 0
I20240502 21:27:27.007159 9367 raft_consensus.cc:482] T d81b71825ba54eacb6c106bce3a838e8 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:27.007359 9367 raft_consensus.cc:504] T d81b71825ba54eacb6c106bce3a838e8 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:27.008190 9367 leader_election.cc:288] T d81b71825ba54eacb6c106bce3a838e8 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:27:27.008872 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81b71825ba54eacb6c106bce3a838e8" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:27:27.009187 4575 raft_consensus.cc:2454] T d81b71825ba54eacb6c106bce3a838e8 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:27.009195 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81b71825ba54eacb6c106bce3a838e8" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:27:27.009436 7214 raft_consensus.cc:2454] T d81b71825ba54eacb6c106bce3a838e8 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:27.009588 7420 leader_election.cc:302] T d81b71825ba54eacb6c106bce3a838e8 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:27:27.010025 9367 raft_consensus.cc:2781] T d81b71825ba54eacb6c106bce3a838e8 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:27.010154 9367 raft_consensus.cc:482] T d81b71825ba54eacb6c106bce3a838e8 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:27.010253 9367 raft_consensus.cc:3037] T d81b71825ba54eacb6c106bce3a838e8 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.012708 9367 raft_consensus.cc:504] T d81b71825ba54eacb6c106bce3a838e8 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:27.013700 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81b71825ba54eacb6c106bce3a838e8" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:27:27.013900 7214 raft_consensus.cc:3037] T d81b71825ba54eacb6c106bce3a838e8 P 39981a8dfe94462b9a73ba489af4c714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.016577 7214 raft_consensus.cc:2454] T d81b71825ba54eacb6c106bce3a838e8 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:27.016978 7419 leader_election.cc:302] T d81b71825ba54eacb6c106bce3a838e8 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:27:27.017424 9225 raft_consensus.cc:2781] T d81b71825ba54eacb6c106bce3a838e8 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:27.017601 9225 raft_consensus.cc:686] T d81b71825ba54eacb6c106bce3a838e8 P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Becoming Leader. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Running, Role: LEADER
I20240502 21:27:27.017884 9225 consensus_queue.cc:237] T d81b71825ba54eacb6c106bce3a838e8 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:27.021629 9367 leader_election.cc:288] T d81b71825ba54eacb6c106bce3a838e8 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Requested vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:27:27.023658 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81b71825ba54eacb6c106bce3a838e8" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:27:27.023972 4575 raft_consensus.cc:3037] T d81b71825ba54eacb6c106bce3a838e8 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.024559 4291 catalog_manager.cc:5484] T d81b71825ba54eacb6c106bce3a838e8 P 31a72286397c4edc9a63eca3412be168 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a72286397c4edc9a63eca3412be168 (127.29.82.71). New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:27.026757 4575 raft_consensus.cc:2454] T d81b71825ba54eacb6c106bce3a838e8 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:27.067220 4291 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:27.075309 9387 tablet.cc:1722] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:27.075918 9387 tablet.cc:1620] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168: MemRowSet was empty: no flush needed.
I20240502 21:27:27.075976 9388 tablet.cc:1722] T 7c2c819c0a274a80b309531af6733885 P 7a736a8ebb164ce0a051c0fccd76c3b8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:27.076514 9388 tablet.cc:1620] T 7c2c819c0a274a80b309531af6733885 P 7a736a8ebb164ce0a051c0fccd76c3b8: MemRowSet was empty: no flush needed.
I20240502 21:27:27.082552 9390 tablet.cc:1722] T 7c2c819c0a274a80b309531af6733885 P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:27.083055 9390 tablet.cc:1620] T 7c2c819c0a274a80b309531af6733885 P 0ddd486109f5434cace13d60d633ac04: MemRowSet was empty: no flush needed.
I20240502 21:27:27.087425 4292 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=6da95202b2c94de5b120f347345ed051] alter complete (version 2)
I20240502 21:27:27.194401 4291 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:27.201738 9387 tablet.cc:1722] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:27.202759 9390 tablet.cc:1722] T 7c2c819c0a274a80b309531af6733885 P 0ddd486109f5434cace13d60d633ac04: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:27.204641 9388 tablet.cc:1722] T 7c2c819c0a274a80b309531af6733885 P 7a736a8ebb164ce0a051c0fccd76c3b8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:27.218835 4291 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=6da95202b2c94de5b120f347345ed051] alter complete (version 3)
I20240502 21:27:27.276882 9036 consensus_queue.cc:1035] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:27.281339 9036 consensus_queue.cc:1035] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:27.328399 4294 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:27.328711 4294 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:27.332255 4294 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet 7c2c819c0a274a80b309531af6733885
I20240502 21:27:27.333091 7597 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c2c819c0a274a80b309531af6733885 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:27 UTC) from {username='slave'} at 127.0.0.1:44444
I20240502 21:27:27.333918 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c2c819c0a274a80b309531af6733885 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:27 UTC) from {username='slave'} at 127.0.0.1:60018
I20240502 21:27:27.334275 7462 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c2c819c0a274a80b309531af6733885 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:27 UTC) from {username='slave'} at 127.0.0.1:37182
I20240502 21:27:27.340071 9399 tablet_replica.cc:330] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:27:27.340378 9399 raft_consensus.cc:2229] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:27.340755 9399 raft_consensus.cc:2258] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:27.342015 9399 ts_tablet_manager.cc:1875] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:27.349560 9398 tablet_replica.cc:330] T 7c2c819c0a274a80b309531af6733885 P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:27:27.349809 9398 raft_consensus.cc:2229] T 7c2c819c0a274a80b309531af6733885 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:27.352689 9397 tablet_replica.cc:330] T 7c2c819c0a274a80b309531af6733885 P 0ddd486109f5434cace13d60d633ac04: stopping tablet replica
I20240502 21:27:27.352844 9399 ts_tablet_manager.cc:1888] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:27.353000 9399 log.cc:1196] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/7c2c819c0a274a80b309531af6733885
I20240502 21:27:27.353528 9399 ts_tablet_manager.cc:1909] T 7c2c819c0a274a80b309531af6733885 P 31a72286397c4edc9a63eca3412be168: Deleting consensus metadata
I20240502 21:27:27.354190 9397 raft_consensus.cc:2229] T 7c2c819c0a274a80b309531af6733885 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:27.354638 9398 raft_consensus.cc:2258] T 7c2c819c0a274a80b309531af6733885 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:27.354681 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet 7c2c819c0a274a80b309531af6733885 (table rebalancer_dynamic_table_new_name [id=6da95202b2c94de5b120f347345ed051]) successfully deleted
I20240502 21:27:27.355304 9397 raft_consensus.cc:2258] T 7c2c819c0a274a80b309531af6733885 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:27.356848 9398 ts_tablet_manager.cc:1875] T 7c2c819c0a274a80b309531af6733885 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:27.357148 9397 ts_tablet_manager.cc:1875] T 7c2c819c0a274a80b309531af6733885 P 0ddd486109f5434cace13d60d633ac04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:27.363279 9398 ts_tablet_manager.cc:1888] T 7c2c819c0a274a80b309531af6733885 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:27.363471 9398 log.cc:1196] T 7c2c819c0a274a80b309531af6733885 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/7c2c819c0a274a80b309531af6733885
I20240502 21:27:27.364082 9398 ts_tablet_manager.cc:1909] T 7c2c819c0a274a80b309531af6733885 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting consensus metadata
I20240502 21:27:27.364272 9397 ts_tablet_manager.cc:1888] T 7c2c819c0a274a80b309531af6733885 P 0ddd486109f5434cace13d60d633ac04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:27.364476 9397 log.cc:1196] T 7c2c819c0a274a80b309531af6733885 P 0ddd486109f5434cace13d60d633ac04: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/wals/7c2c819c0a274a80b309531af6733885
I20240502 21:27:27.365123 9397 ts_tablet_manager.cc:1909] T 7c2c819c0a274a80b309531af6733885 P 0ddd486109f5434cace13d60d633ac04: Deleting consensus metadata
I20240502 21:27:27.365878 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet 7c2c819c0a274a80b309531af6733885 (table rebalancer_dynamic_table_new_name [id=6da95202b2c94de5b120f347345ed051]) successfully deleted
I20240502 21:27:27.366560 4285 catalog_manager.cc:4830] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): tablet 7c2c819c0a274a80b309531af6733885 (table rebalancer_dynamic_table_new_name [id=6da95202b2c94de5b120f347345ed051]) successfully deleted
I20240502 21:27:27.435117 4294 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:27.445255 4817 tablet_service.cc:1467] Processing CreateTablet for tablet b536bbad7d7948a5a3e71dce41569b20 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bb7ebe335b2b49a8b313d705a6d3b4d5]), partition=
I20240502 21:27:27.445333 4686 tablet_service.cc:1467] Processing CreateTablet for tablet b536bbad7d7948a5a3e71dce41569b20 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bb7ebe335b2b49a8b313d705a6d3b4d5]), partition=
I20240502 21:27:27.445636 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b536bbad7d7948a5a3e71dce41569b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:27.445700 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b536bbad7d7948a5a3e71dce41569b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:27.451185 9401 tablet_bootstrap.cc:492] T b536bbad7d7948a5a3e71dce41569b20 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:27.451198 4422 tablet_service.cc:1467] Processing CreateTablet for tablet b536bbad7d7948a5a3e71dce41569b20 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bb7ebe335b2b49a8b313d705a6d3b4d5]), partition=
I20240502 21:27:27.451562 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b536bbad7d7948a5a3e71dce41569b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:27.451707 9400 tablet_bootstrap.cc:492] T b536bbad7d7948a5a3e71dce41569b20 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:27.452898 9401 tablet_bootstrap.cc:654] T b536bbad7d7948a5a3e71dce41569b20 P 398c3c8f3195492c900439ad5e1931bb: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:27.452917 9400 tablet_bootstrap.cc:654] T b536bbad7d7948a5a3e71dce41569b20 P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:27.455830 9401 tablet_bootstrap.cc:492] T b536bbad7d7948a5a3e71dce41569b20 P 398c3c8f3195492c900439ad5e1931bb: No bootstrap required, opened a new log
I20240502 21:27:27.456053 9401 ts_tablet_manager.cc:1367] T b536bbad7d7948a5a3e71dce41569b20 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:27:27.456888 9401 raft_consensus.cc:348] T b536bbad7d7948a5a3e71dce41569b20 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:27.457247 9401 raft_consensus.cc:374] T b536bbad7d7948a5a3e71dce41569b20 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:27.457410 9401 raft_consensus.cc:729] T b536bbad7d7948a5a3e71dce41569b20 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:27.458742 9401 consensus_queue.cc:260] T b536bbad7d7948a5a3e71dce41569b20 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:27.459271 9401 ts_tablet_manager.cc:1398] T b536bbad7d7948a5a3e71dce41569b20 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:27.460662 9404 tablet_bootstrap.cc:492] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:27.461012 9400 tablet_bootstrap.cc:492] T b536bbad7d7948a5a3e71dce41569b20 P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:27.461158 9400 ts_tablet_manager.cc:1367] T b536bbad7d7948a5a3e71dce41569b20 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20240502 21:27:27.461853 9404 tablet_bootstrap.cc:654] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:27.463310 9400 raft_consensus.cc:348] T b536bbad7d7948a5a3e71dce41569b20 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:27.463516 9400 raft_consensus.cc:374] T b536bbad7d7948a5a3e71dce41569b20 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:27.463649 9400 raft_consensus.cc:729] T b536bbad7d7948a5a3e71dce41569b20 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:27.463908 9400 consensus_queue.cc:260] T b536bbad7d7948a5a3e71dce41569b20 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:27.466013 9404 tablet_bootstrap.cc:492] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:27.466157 9404 ts_tablet_manager.cc:1367] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240502 21:27:27.466801 9404 raft_consensus.cc:348] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:27.467010 9404 raft_consensus.cc:374] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:27.467082 9404 raft_consensus.cc:729] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:27.467345 9404 consensus_queue.cc:260] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:27.467861 9404 ts_tablet_manager.cc:1398] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:27.468163 9400 ts_tablet_manager.cc:1398] T b536bbad7d7948a5a3e71dce41569b20 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 21:27:27.475560 9314 consensus_queue.cc:1035] T 37c2c46368144212a0041e7ff4434236 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:27.479827 9314 consensus_queue.cc:1035] T 37c2c46368144212a0041e7ff4434236 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:27.568094 9367 consensus_queue.cc:1035] T d81b71825ba54eacb6c106bce3a838e8 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:27.573664 9140 consensus_queue.cc:1035] T d81b71825ba54eacb6c106bce3a838e8 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:27.585258 4294 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_extra_table_64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:27.596104 7328 tablet_service.cc:1467] Processing CreateTablet for tablet 25be332e341d4869b2165da105c17a8a (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=8bc8342ce686477cac74df1ebaa3656a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:27.596542 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25be332e341d4869b2165da105c17a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:27.599709 4422 tablet_service.cc:1467] Processing CreateTablet for tablet 25be332e341d4869b2165da105c17a8a (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=8bc8342ce686477cac74df1ebaa3656a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:27.600165 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25be332e341d4869b2165da105c17a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:27.601109 7597 tablet_service.cc:1467] Processing CreateTablet for tablet 25be332e341d4869b2165da105c17a8a (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=8bc8342ce686477cac74df1ebaa3656a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:27.601524 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25be332e341d4869b2165da105c17a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:27.601642 4555 tablet_service.cc:1467] Processing CreateTablet for tablet 402e76e6f92744fbb36653d04f861e4f (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=8bc8342ce686477cac74df1ebaa3656a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:27.601981 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 402e76e6f92744fbb36653d04f861e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:27.606128 9404 tablet_bootstrap.cc:492] T 25be332e341d4869b2165da105c17a8a P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:27.607596 9404 tablet_bootstrap.cc:654] T 25be332e341d4869b2165da105c17a8a P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:27.608860 7462 tablet_service.cc:1467] Processing CreateTablet for tablet 402e76e6f92744fbb36653d04f861e4f (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=8bc8342ce686477cac74df1ebaa3656a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:27.609334 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 402e76e6f92744fbb36653d04f861e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:27.609535 4421 tablet_service.cc:1467] Processing CreateTablet for tablet 402e76e6f92744fbb36653d04f861e4f (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=8bc8342ce686477cac74df1ebaa3656a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:27.609918 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 402e76e6f92744fbb36653d04f861e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:27.611845 4554 tablet_service.cc:1467] Processing CreateTablet for tablet 59ef00d58c244b40ba4eebe040f8ffbd (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=8bc8342ce686477cac74df1ebaa3656a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:27.612300 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59ef00d58c244b40ba4eebe040f8ffbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:27.616360 7329 tablet_service.cc:1467] Processing CreateTablet for tablet 59ef00d58c244b40ba4eebe040f8ffbd (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=8bc8342ce686477cac74df1ebaa3656a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:27.616824 7329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59ef00d58c244b40ba4eebe040f8ffbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:27.619781 4817 tablet_service.cc:1467] Processing CreateTablet for tablet 59ef00d58c244b40ba4eebe040f8ffbd (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=8bc8342ce686477cac74df1ebaa3656a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:27.620203 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59ef00d58c244b40ba4eebe040f8ffbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:27.629379 9412 tablet_bootstrap.cc:492] T 402e76e6f92744fbb36653d04f861e4f P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:27.633579 9415 tablet_bootstrap.cc:492] T 25be332e341d4869b2165da105c17a8a P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:27.634516 9412 tablet_bootstrap.cc:654] T 402e76e6f92744fbb36653d04f861e4f P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:27.636288 9400 tablet_bootstrap.cc:492] T 59ef00d58c244b40ba4eebe040f8ffbd P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:27.637429 9404 tablet_bootstrap.cc:492] T 25be332e341d4869b2165da105c17a8a P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:27.637615 9404 ts_tablet_manager.cc:1367] T 25be332e341d4869b2165da105c17a8a P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.001s
I20240502 21:27:27.637804 9400 tablet_bootstrap.cc:654] T 59ef00d58c244b40ba4eebe040f8ffbd P 73a1f2ca1f544de9b0e3b37015f85328: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:27.638517 9404 raft_consensus.cc:348] T 25be332e341d4869b2165da105c17a8a P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:27.638811 9404 raft_consensus.cc:374] T 25be332e341d4869b2165da105c17a8a P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:27.638875 9404 raft_consensus.cc:729] T 25be332e341d4869b2165da105c17a8a P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:27.639861 9404 consensus_queue.cc:260] T 25be332e341d4869b2165da105c17a8a P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:27.641641 9404 ts_tablet_manager.cc:1398] T 25be332e341d4869b2165da105c17a8a P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:27:27.643113 9404 tablet_bootstrap.cc:492] T 402e76e6f92744fbb36653d04f861e4f P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:27.643692 9416 tablet_bootstrap.cc:492] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:27.644299 9412 tablet_bootstrap.cc:492] T 402e76e6f92744fbb36653d04f861e4f P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:27.644495 9412 ts_tablet_manager.cc:1367] T 402e76e6f92744fbb36653d04f861e4f P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20240502 21:27:27.644650 9404 tablet_bootstrap.cc:654] T 402e76e6f92744fbb36653d04f861e4f P 71e1375b799b439fa3c83ca763c039c1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:27.646083 9412 raft_consensus.cc:348] T 402e76e6f92744fbb36653d04f861e4f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:27.646365 9400 tablet_bootstrap.cc:492] T 59ef00d58c244b40ba4eebe040f8ffbd P 73a1f2ca1f544de9b0e3b37015f85328: No bootstrap required, opened a new log
I20240502 21:27:27.646364 9412 raft_consensus.cc:374] T 402e76e6f92744fbb36653d04f861e4f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:27.646490 9412 raft_consensus.cc:729] T 402e76e6f92744fbb36653d04f861e4f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:27.646880 9412 consensus_queue.cc:260] T 402e76e6f92744fbb36653d04f861e4f P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:27.647264 9415 tablet_bootstrap.cc:654] T 25be332e341d4869b2165da105c17a8a P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:27.647262 9400 ts_tablet_manager.cc:1367] T 59ef00d58c244b40ba4eebe040f8ffbd P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20240502 21:27:27.647723 9412 ts_tablet_manager.cc:1398] T 402e76e6f92744fbb36653d04f861e4f P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:27:27.648319 9412 tablet_bootstrap.cc:492] T 59ef00d58c244b40ba4eebe040f8ffbd P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:27.648614 9400 raft_consensus.cc:348] T 59ef00d58c244b40ba4eebe040f8ffbd P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:27.648887 9400 raft_consensus.cc:374] T 59ef00d58c244b40ba4eebe040f8ffbd P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:27.648989 9400 raft_consensus.cc:729] T 59ef00d58c244b40ba4eebe040f8ffbd P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:27.649318 9400 consensus_queue.cc:260] T 59ef00d58c244b40ba4eebe040f8ffbd P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:27.650095 9400 ts_tablet_manager.cc:1398] T 59ef00d58c244b40ba4eebe040f8ffbd P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:27.651528 9404 tablet_bootstrap.cc:492] T 402e76e6f92744fbb36653d04f861e4f P 71e1375b799b439fa3c83ca763c039c1: No bootstrap required, opened a new log
I20240502 21:27:27.651760 9404 ts_tablet_manager.cc:1367] T 402e76e6f92744fbb36653d04f861e4f P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240502 21:27:27.653208 9412 tablet_bootstrap.cc:654] T 59ef00d58c244b40ba4eebe040f8ffbd P 7a736a8ebb164ce0a051c0fccd76c3b8: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:27.654126 9404 raft_consensus.cc:348] T 402e76e6f92744fbb36653d04f861e4f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:27.654402 9404 raft_consensus.cc:374] T 402e76e6f92744fbb36653d04f861e4f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:27.654517 9404 raft_consensus.cc:729] T 402e76e6f92744fbb36653d04f861e4f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:27.654806 9416 tablet_bootstrap.cc:654] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:27.654826 9404 consensus_queue.cc:260] T 402e76e6f92744fbb36653d04f861e4f P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:27.655388 9404 ts_tablet_manager.cc:1398] T 402e76e6f92744fbb36653d04f861e4f P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:27:27.664120 9413 tablet_bootstrap.cc:492] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:27.665355 9412 tablet_bootstrap.cc:492] T 59ef00d58c244b40ba4eebe040f8ffbd P 7a736a8ebb164ce0a051c0fccd76c3b8: No bootstrap required, opened a new log
I20240502 21:27:27.665540 9412 ts_tablet_manager.cc:1367] T 59ef00d58c244b40ba4eebe040f8ffbd P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20240502 21:27:27.666190 9413 tablet_bootstrap.cc:654] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:27.666364 9415 tablet_bootstrap.cc:492] T 25be332e341d4869b2165da105c17a8a P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:27.666375 9412 raft_consensus.cc:348] T 59ef00d58c244b40ba4eebe040f8ffbd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:27.666539 9415 ts_tablet_manager.cc:1367] T 25be332e341d4869b2165da105c17a8a P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.002s
I20240502 21:27:27.666702 9412 raft_consensus.cc:374] T 59ef00d58c244b40ba4eebe040f8ffbd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:27.666842 9412 raft_consensus.cc:729] T 59ef00d58c244b40ba4eebe040f8ffbd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:27.667259 9416 tablet_bootstrap.cc:492] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:27.667165 9412 consensus_queue.cc:260] T 59ef00d58c244b40ba4eebe040f8ffbd P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:27.667421 9416 ts_tablet_manager.cc:1367] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.006s
I20240502 21:27:27.668268 9416 raft_consensus.cc:348] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:27.668495 9416 raft_consensus.cc:374] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:27.669490 9412 ts_tablet_manager.cc:1398] T 59ef00d58c244b40ba4eebe040f8ffbd P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 21:27:27.669387 9415 raft_consensus.cc:348] T 25be332e341d4869b2165da105c17a8a P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:27.669750 9416 raft_consensus.cc:729] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:27.669960 9415 raft_consensus.cc:374] T 25be332e341d4869b2165da105c17a8a P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:27.670238 9353 raft_consensus.cc:482] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:27.670388 9353 raft_consensus.cc:504] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:27.671008 9353 leader_election.cc:288] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:27:27.671388 9415 raft_consensus.cc:729] T 25be332e341d4869b2165da105c17a8a P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:27.671525 4837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b536bbad7d7948a5a3e71dce41569b20" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:27:27.671551 9416 consensus_queue.cc:260] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:27.671765 4837 raft_consensus.cc:2454] T b536bbad7d7948a5a3e71dce41569b20 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:27.671764 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b536bbad7d7948a5a3e71dce41569b20" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb" is_pre_election: true
I20240502 21:27:27.672016 4705 raft_consensus.cc:2454] T b536bbad7d7948a5a3e71dce41569b20 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 0.
I20240502 21:27:27.672370 4375 leader_election.cc:302] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e1375b799b439fa3c83ca763c039c1, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:27:27.672694 9415 consensus_queue.cc:260] T 25be332e341d4869b2165da105c17a8a P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:27.672989 9036 raft_consensus.cc:2781] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:27.673072 9036 raft_consensus.cc:482] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:27.673141 9036 raft_consensus.cc:3037] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.673432 9416 ts_tablet_manager.cc:1398] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 21:27:27.674065 9415 ts_tablet_manager.cc:1398] T 25be332e341d4869b2165da105c17a8a P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 21:27:27.674358 9415 tablet_bootstrap.cc:492] T 59ef00d58c244b40ba4eebe040f8ffbd P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:27.675835 9415 tablet_bootstrap.cc:654] T 59ef00d58c244b40ba4eebe040f8ffbd P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:27.675545 9036 raft_consensus.cc:504] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:27.681370 9036 leader_election.cc:288] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Requested vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:27:27.682168 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b536bbad7d7948a5a3e71dce41569b20" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:27:27.682473 4705 raft_consensus.cc:3037] T b536bbad7d7948a5a3e71dce41569b20 P 398c3c8f3195492c900439ad5e1931bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.682559 4837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b536bbad7d7948a5a3e71dce41569b20" candidate_uuid: "71e1375b799b439fa3c83ca763c039c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:27:27.682780 4837 raft_consensus.cc:3037] T b536bbad7d7948a5a3e71dce41569b20 P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.686542 4705 raft_consensus.cc:2454] T b536bbad7d7948a5a3e71dce41569b20 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:27.687170 4378 leader_election.cc:302] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:27.687587 9036 raft_consensus.cc:2781] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:27.687860 9036 raft_consensus.cc:686] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Running, Role: LEADER
I20240502 21:27:27.688323 9036 consensus_queue.cc:237] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:27.688601 4837 raft_consensus.cc:2454] T b536bbad7d7948a5a3e71dce41569b20 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e1375b799b439fa3c83ca763c039c1 in term 1.
I20240502 21:27:27.698187 9413 tablet_bootstrap.cc:492] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:27.697957 4294 catalog_manager.cc:5484] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65). New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:27.698884 9413 ts_tablet_manager.cc:1367] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.008s
I20240502 21:27:27.700249 9413 raft_consensus.cc:348] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:27.700484 9413 raft_consensus.cc:374] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:27.700587 9413 raft_consensus.cc:729] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:27.700848 9413 consensus_queue.cc:260] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:27.702812 9413 ts_tablet_manager.cc:1398] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:27.704268 9415 tablet_bootstrap.cc:492] T 59ef00d58c244b40ba4eebe040f8ffbd P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:27.704509 9415 ts_tablet_manager.cc:1367] T 59ef00d58c244b40ba4eebe040f8ffbd P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.030s user 0.015s sys 0.012s
I20240502 21:27:27.705240 9415 raft_consensus.cc:348] T 59ef00d58c244b40ba4eebe040f8ffbd P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:27.705408 9415 raft_consensus.cc:374] T 59ef00d58c244b40ba4eebe040f8ffbd P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:27.705483 9415 raft_consensus.cc:729] T 59ef00d58c244b40ba4eebe040f8ffbd P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:27.705708 9415 consensus_queue.cc:260] T 59ef00d58c244b40ba4eebe040f8ffbd P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:27.707118 9415 ts_tablet_manager.cc:1398] T 59ef00d58c244b40ba4eebe040f8ffbd P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:27.764017 9140 raft_consensus.cc:482] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:27.764217 9140 raft_consensus.cc:504] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:27.764951 9140 leader_election.cc:288] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:27.765480 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "402e76e6f92744fbb36653d04f861e4f" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:27:27.765588 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "402e76e6f92744fbb36653d04f861e4f" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:27:27.765726 4440 raft_consensus.cc:2454] T 402e76e6f92744fbb36653d04f861e4f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:27.765846 4574 raft_consensus.cc:2454] T 402e76e6f92744fbb36653d04f861e4f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 0.
I20240502 21:27:27.766360 7416 leader_election.cc:302] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:27.766861 9140 raft_consensus.cc:2781] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:27.766952 9140 raft_consensus.cc:482] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:27.767067 9140 raft_consensus.cc:3037] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.770169 9140 raft_consensus.cc:504] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:27.772941 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "402e76e6f92744fbb36653d04f861e4f" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:27:27.773156 4440 raft_consensus.cc:3037] T 402e76e6f92744fbb36653d04f861e4f P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.778700 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "402e76e6f92744fbb36653d04f861e4f" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:27:27.778954 4574 raft_consensus.cc:3037] T 402e76e6f92744fbb36653d04f861e4f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.779304 4440 raft_consensus.cc:2454] T 402e76e6f92744fbb36653d04f861e4f P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:27.779721 7416 leader_election.cc:302] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:27:27.780030 9367 raft_consensus.cc:2781] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:27.780200 9367 raft_consensus.cc:686] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Becoming Leader. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Running, Role: LEADER
I20240502 21:27:27.780499 9367 consensus_queue.cc:237] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:27.780838 9071 raft_consensus.cc:482] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:27.781020 9071 raft_consensus.cc:504] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:27.781754 9140 leader_election.cc:288] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 1 election: Requested vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:27.781880 9071 leader_election.cc:288] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:27.785527 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25be332e341d4869b2165da105c17a8a" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:27.785836 7350 raft_consensus.cc:2454] T 25be332e341d4869b2165da105c17a8a P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:27.788535 7551 leader_election.cc:302] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:27.789716 9071 raft_consensus.cc:2781] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:27.789837 9071 raft_consensus.cc:482] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:27.789934 9071 raft_consensus.cc:3037] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.790163 4292 catalog_manager.cc:5484] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a72286397c4edc9a63eca3412be168 (127.29.82.71). New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:27.792970 4574 raft_consensus.cc:2454] T 402e76e6f92744fbb36653d04f861e4f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:27:27.812605 9071 raft_consensus.cc:504] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:27.813771 7348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25be332e341d4869b2165da105c17a8a" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:27.814056 7348 raft_consensus.cc:3037] T 25be332e341d4869b2165da105c17a8a P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.814998 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25be332e341d4869b2165da105c17a8a" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1" is_pre_election: true
I20240502 21:27:27.815232 4440 raft_consensus.cc:2454] T 25be332e341d4869b2165da105c17a8a P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:27.817155 9071 leader_election.cc:288] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:27.820930 7348 raft_consensus.cc:2454] T 25be332e341d4869b2165da105c17a8a P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:27.823207 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:27.828449 7551 leader_election.cc:302] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:27.828855 9071 raft_consensus.cc:2781] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:27.829039 9071 raft_consensus.cc:686] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Becoming Leader. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Running, Role: LEADER
I20240502 21:27:27.829351 9071 consensus_queue.cc:237] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:27.835254 4298 catalog_manager.cc:5484] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ddd486109f5434cace13d60d633ac04 (127.29.82.72). New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:27.841058 4705 raft_consensus.cc:1261] T b536bbad7d7948a5a3e71dce41569b20 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:27.841712 9036 consensus_queue.cc:1035] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:27.847512 4442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25be332e341d4869b2165da105c17a8a" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:27:27.847760 4442 raft_consensus.cc:3037] T 25be332e341d4869b2165da105c17a8a P 71e1375b799b439fa3c83ca763c039c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.848510 4837 raft_consensus.cc:1261] T b536bbad7d7948a5a3e71dce41569b20 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:27.849803 4292 catalog_manager.cc:5576] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537) does not have the latest schema for tablet b536bbad7d7948a5a3e71dce41569b20 (table rebalancer_dynamic_table [id=bb7ebe335b2b49a8b313d705a6d3b4d5]). Expected version 1 got 0
I20240502 21:27:27.850512 4442 raft_consensus.cc:2454] T 25be332e341d4869b2165da105c17a8a P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:27.852154 9353 consensus_queue.cc:1035] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:27.854223 9431 tablet.cc:1722] T b536bbad7d7948a5a3e71dce41569b20 P 398c3c8f3195492c900439ad5e1931bb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:27.855044 9431 tablet.cc:1620] T b536bbad7d7948a5a3e71dce41569b20 P 398c3c8f3195492c900439ad5e1931bb: MemRowSet was empty: no flush needed.
I20240502 21:27:27.858793 4292 catalog_manager.cc:5576] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537) does not have the latest schema for tablet b536bbad7d7948a5a3e71dce41569b20 (table rebalancer_dynamic_table [id=bb7ebe335b2b49a8b313d705a6d3b4d5]). Expected version 1 got 0
I20240502 21:27:27.865188 9430 tablet.cc:1722] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:27.865693 9430 tablet.cc:1620] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:27.881608 4294 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=bb7ebe335b2b49a8b313d705a6d3b4d5] alter complete (version 1)
I20240502 21:27:27.892788 9434 tablet.cc:1722] T b536bbad7d7948a5a3e71dce41569b20 P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:27.893846 9434 tablet.cc:1620] T b536bbad7d7948a5a3e71dce41569b20 P 73a1f2ca1f544de9b0e3b37015f85328: MemRowSet was empty: no flush needed.
I20240502 21:27:27.896370 4294 catalog_manager.cc:5576] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449) does not have the latest schema for tablet b536bbad7d7948a5a3e71dce41569b20 (table rebalancer_dynamic_table [id=bb7ebe335b2b49a8b313d705a6d3b4d5]). Expected version 1 got 0
I20240502 21:27:27.927793 9272 raft_consensus.cc:482] T 59ef00d58c244b40ba4eebe040f8ffbd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:27.927997 9272 raft_consensus.cc:504] T 59ef00d58c244b40ba4eebe040f8ffbd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:27.928787 9272 leader_election.cc:288] T 59ef00d58c244b40ba4eebe040f8ffbd P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:27.929286 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59ef00d58c244b40ba4eebe040f8ffbd" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:27.929392 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59ef00d58c244b40ba4eebe040f8ffbd" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:27:27.929570 7350 raft_consensus.cc:2454] T 59ef00d58c244b40ba4eebe040f8ffbd P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:27:27.929642 4839 raft_consensus.cc:2454] T 59ef00d58c244b40ba4eebe040f8ffbd P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 0.
I20240502 21:27:27.930004 4510 leader_election.cc:302] T 59ef00d58c244b40ba4eebe040f8ffbd P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a736a8ebb164ce0a051c0fccd76c3b8, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:27.930248 9272 raft_consensus.cc:2781] T 59ef00d58c244b40ba4eebe040f8ffbd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:27.930330 9272 raft_consensus.cc:482] T 59ef00d58c244b40ba4eebe040f8ffbd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:27.930403 9272 raft_consensus.cc:3037] T 59ef00d58c244b40ba4eebe040f8ffbd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.932363 9272 raft_consensus.cc:504] T 59ef00d58c244b40ba4eebe040f8ffbd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:27.932979 9272 leader_election.cc:288] T 59ef00d58c244b40ba4eebe040f8ffbd P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Requested vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:27.933508 7348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59ef00d58c244b40ba4eebe040f8ffbd" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:27.933642 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59ef00d58c244b40ba4eebe040f8ffbd" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:27:27.933729 7348 raft_consensus.cc:3037] T 59ef00d58c244b40ba4eebe040f8ffbd P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.933833 4839 raft_consensus.cc:3037] T 59ef00d58c244b40ba4eebe040f8ffbd P 73a1f2ca1f544de9b0e3b37015f85328 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.936123 4839 raft_consensus.cc:2454] T 59ef00d58c244b40ba4eebe040f8ffbd P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:27:27.936139 7348 raft_consensus.cc:2454] T 59ef00d58c244b40ba4eebe040f8ffbd P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:27:27.936483 4510 leader_election.cc:302] T 59ef00d58c244b40ba4eebe040f8ffbd P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a736a8ebb164ce0a051c0fccd76c3b8, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:27.936810 9272 raft_consensus.cc:2781] T 59ef00d58c244b40ba4eebe040f8ffbd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:27.937179 9272 raft_consensus.cc:686] T 59ef00d58c244b40ba4eebe040f8ffbd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:27:27.937719 9272 consensus_queue.cc:237] T 59ef00d58c244b40ba4eebe040f8ffbd P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:27.942523 4294 catalog_manager.cc:5484] T 59ef00d58c244b40ba4eebe040f8ffbd P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:27.990001 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:27.997555 9430 tablet.cc:1722] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:27.998072 9430 tablet.cc:1620] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1: MemRowSet was empty: no flush needed.
I20240502 21:27:27.998327 9434 tablet.cc:1722] T b536bbad7d7948a5a3e71dce41569b20 P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:27.998823 9434 tablet.cc:1620] T b536bbad7d7948a5a3e71dce41569b20 P 73a1f2ca1f544de9b0e3b37015f85328: MemRowSet was empty: no flush needed.
I20240502 21:27:27.999001 9431 tablet.cc:1722] T b536bbad7d7948a5a3e71dce41569b20 P 398c3c8f3195492c900439ad5e1931bb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:27.999440 9431 tablet.cc:1620] T b536bbad7d7948a5a3e71dce41569b20 P 398c3c8f3195492c900439ad5e1931bb: MemRowSet was empty: no flush needed.
I20240502 21:27:28.009508 4292 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table [id=bb7ebe335b2b49a8b313d705a6d3b4d5] alter complete (version 2)
I20240502 21:27:28.112753 4294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:28.121996 9430 tablet.cc:1722] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:28.123298 9431 tablet.cc:1722] T b536bbad7d7948a5a3e71dce41569b20 P 398c3c8f3195492c900439ad5e1931bb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:28.126313 9434 tablet.cc:1722] T b536bbad7d7948a5a3e71dce41569b20 P 73a1f2ca1f544de9b0e3b37015f85328: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:28.133862 4294 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: rebalancer_dynamic_table_new_name [id=bb7ebe335b2b49a8b313d705a6d3b4d5] alter complete (version 3)
I20240502 21:27:28.239135 4291 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:28.239354 4291 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33456:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:28.242416 4291 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e4b7d987f0ff4a3589447bd47c1ddd44: Sending DeleteTablet for 3 replicas of tablet b536bbad7d7948a5a3e71dce41569b20
I20240502 21:27:28.243474 4817 tablet_service.cc:1514] Processing DeleteTablet for tablet b536bbad7d7948a5a3e71dce41569b20 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:28 UTC) from {username='slave'} at 127.0.0.1:56326
I20240502 21:27:28.243822 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet b536bbad7d7948a5a3e71dce41569b20 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:28 UTC) from {username='slave'} at 127.0.0.1:44764
I20240502 21:27:28.244086 9438 tablet_replica.cc:330] T b536bbad7d7948a5a3e71dce41569b20 P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:27:28.244451 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet b536bbad7d7948a5a3e71dce41569b20 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:28 UTC) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:28.244962 9439 tablet_replica.cc:330] T b536bbad7d7948a5a3e71dce41569b20 P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:27:28.245221 9439 raft_consensus.cc:2229] T b536bbad7d7948a5a3e71dce41569b20 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:28.245527 9439 raft_consensus.cc:2258] T b536bbad7d7948a5a3e71dce41569b20 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:28.246892 9439 ts_tablet_manager.cc:1875] T b536bbad7d7948a5a3e71dce41569b20 P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:28.247514 9438 raft_consensus.cc:2229] T b536bbad7d7948a5a3e71dce41569b20 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:28.247809 9438 raft_consensus.cc:2258] T b536bbad7d7948a5a3e71dce41569b20 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:28.248849 9438 ts_tablet_manager.cc:1875] T b536bbad7d7948a5a3e71dce41569b20 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:28.250592 9440 tablet_replica.cc:330] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:28.251715 9440 raft_consensus.cc:2229] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:28.252202 9440 raft_consensus.cc:2258] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:28.253223 9440 ts_tablet_manager.cc:1875] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:28.256155 9438 ts_tablet_manager.cc:1888] T b536bbad7d7948a5a3e71dce41569b20 P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:28.256336 9438 log.cc:1196] T b536bbad7d7948a5a3e71dce41569b20 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/b536bbad7d7948a5a3e71dce41569b20
I20240502 21:27:28.256719 9439 ts_tablet_manager.cc:1888] T b536bbad7d7948a5a3e71dce41569b20 P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:28.256899 9439 log.cc:1196] T b536bbad7d7948a5a3e71dce41569b20 P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/b536bbad7d7948a5a3e71dce41569b20
I20240502 21:27:28.257367 9439 ts_tablet_manager.cc:1909] T b536bbad7d7948a5a3e71dce41569b20 P 398c3c8f3195492c900439ad5e1931bb: Deleting consensus metadata
I20240502 21:27:28.257469 9438 ts_tablet_manager.cc:1909] T b536bbad7d7948a5a3e71dce41569b20 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting consensus metadata
I20240502 21:27:28.258446 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet b536bbad7d7948a5a3e71dce41569b20 (table rebalancer_dynamic_table_new_name [id=bb7ebe335b2b49a8b313d705a6d3b4d5]) successfully deleted
I20240502 21:27:28.259231 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet b536bbad7d7948a5a3e71dce41569b20 (table rebalancer_dynamic_table_new_name [id=bb7ebe335b2b49a8b313d705a6d3b4d5]) successfully deleted
I20240502 21:27:28.260825 9440 ts_tablet_manager.cc:1888] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:28.260941 9440 log.cc:1196] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/b536bbad7d7948a5a3e71dce41569b20
I20240502 21:27:28.261314 9440 ts_tablet_manager.cc:1909] T b536bbad7d7948a5a3e71dce41569b20 P 71e1375b799b439fa3c83ca763c039c1: Deleting consensus metadata
I20240502 21:27:28.262267 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet b536bbad7d7948a5a3e71dce41569b20 (table rebalancer_dynamic_table_new_name [id=bb7ebe335b2b49a8b313d705a6d3b4d5]) successfully deleted
I20240502 21:27:28.328881 9367 consensus_queue.cc:1035] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:28.342732 9140 consensus_queue.cc:1035] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:28.353847 4291 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33456:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:28.366425 7462 tablet_service.cc:1467] Processing CreateTablet for tablet d558bcaf20bd46548936d5fb946d7710 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=52a205feacb541d3a82d26d91459b07c]), partition=
I20240502 21:27:28.366813 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d558bcaf20bd46548936d5fb946d7710. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.367556 7328 tablet_service.cc:1467] Processing CreateTablet for tablet d558bcaf20bd46548936d5fb946d7710 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=52a205feacb541d3a82d26d91459b07c]), partition=
I20240502 21:27:28.367936 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d558bcaf20bd46548936d5fb946d7710. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.372082 9446 tablet_bootstrap.cc:492] T d558bcaf20bd46548936d5fb946d7710 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:28.373854 9446 tablet_bootstrap.cc:654] T d558bcaf20bd46548936d5fb946d7710 P 31a72286397c4edc9a63eca3412be168: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:28.379884 7597 tablet_service.cc:1467] Processing CreateTablet for tablet d558bcaf20bd46548936d5fb946d7710 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=52a205feacb541d3a82d26d91459b07c]), partition=
I20240502 21:27:28.380298 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d558bcaf20bd46548936d5fb946d7710. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.403141 9450 tablet_bootstrap.cc:492] T d558bcaf20bd46548936d5fb946d7710 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:28.405416 9448 tablet_bootstrap.cc:492] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:28.407342 9448 tablet_bootstrap.cc:654] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:28.407418 9450 tablet_bootstrap.cc:654] T d558bcaf20bd46548936d5fb946d7710 P d0456c5e55a542128688904e003d403c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:28.410557 9272 consensus_queue.cc:1035] T 59ef00d58c244b40ba4eebe040f8ffbd P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:28.420316 9448 tablet_bootstrap.cc:492] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04: No bootstrap required, opened a new log
I20240502 21:27:28.420482 9448 ts_tablet_manager.cc:1367] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.003s
I20240502 21:27:28.421255 9448 raft_consensus.cc:348] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:28.421471 9448 raft_consensus.cc:374] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:28.421569 9448 raft_consensus.cc:729] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:28.421803 9448 consensus_queue.cc:260] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:28.422376 9448 ts_tablet_manager.cc:1398] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:27:28.422397 9450 tablet_bootstrap.cc:492] T d558bcaf20bd46548936d5fb946d7710 P d0456c5e55a542128688904e003d403c: No bootstrap required, opened a new log
I20240502 21:27:28.422606 9450 ts_tablet_manager.cc:1367] T d558bcaf20bd46548936d5fb946d7710 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.005s
I20240502 21:27:28.423871 9450 raft_consensus.cc:348] T d558bcaf20bd46548936d5fb946d7710 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:28.424249 9450 raft_consensus.cc:374] T d558bcaf20bd46548936d5fb946d7710 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:28.424438 9450 raft_consensus.cc:729] T d558bcaf20bd46548936d5fb946d7710 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:28.424849 9450 consensus_queue.cc:260] T d558bcaf20bd46548936d5fb946d7710 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:28.425937 9450 ts_tablet_manager.cc:1398] T d558bcaf20bd46548936d5fb946d7710 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:28.429454 9272 consensus_queue.cc:1035] T 59ef00d58c244b40ba4eebe040f8ffbd P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:28.444535 9071 consensus_queue.cc:1035] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:28.447407 4641 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.70:40719, user_credentials={real_user=slave}} blocked reactor thread for 82254us
I20240502 21:27:28.458364 9446 tablet_bootstrap.cc:492] T d558bcaf20bd46548936d5fb946d7710 P 31a72286397c4edc9a63eca3412be168: No bootstrap required, opened a new log
I20240502 21:27:28.458571 9446 ts_tablet_manager.cc:1367] T d558bcaf20bd46548936d5fb946d7710 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.087s user 0.021s sys 0.052s
I20240502 21:27:28.459470 9446 raft_consensus.cc:348] T d558bcaf20bd46548936d5fb946d7710 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:28.459483 9457 consensus_queue.cc:1035] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:28.459813 9446 raft_consensus.cc:374] T d558bcaf20bd46548936d5fb946d7710 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:28.459967 9446 raft_consensus.cc:729] T d558bcaf20bd46548936d5fb946d7710 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:28.460289 9446 consensus_queue.cc:260] T d558bcaf20bd46548936d5fb946d7710 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:28.461031 9446 ts_tablet_manager.cc:1398] T d558bcaf20bd46548936d5fb946d7710 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:28.721918 9071 raft_consensus.cc:482] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:28.722098 9071 raft_consensus.cc:504] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:28.722790 9071 leader_election.cc:288] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:28.723194 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d558bcaf20bd46548936d5fb946d7710" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:27:28.723245 7482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d558bcaf20bd46548936d5fb946d7710" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:27:28.723436 7482 raft_consensus.cc:2454] T d558bcaf20bd46548936d5fb946d7710 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:28.723431 7350 raft_consensus.cc:2454] T d558bcaf20bd46548936d5fb946d7710 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 0.
I20240502 21:27:28.723887 7551 leader_election.cc:302] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:28.724215 9071 raft_consensus.cc:2781] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:28.724347 9071 raft_consensus.cc:482] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:28.724450 9071 raft_consensus.cc:3037] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:28.727319 9071 raft_consensus.cc:504] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:28.728406 9071 leader_election.cc:288] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:28.728742 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d558bcaf20bd46548936d5fb946d7710" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:28.728981 7350 raft_consensus.cc:3037] T d558bcaf20bd46548936d5fb946d7710 P d0456c5e55a542128688904e003d403c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:28.728931 7482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d558bcaf20bd46548936d5fb946d7710" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:28.729151 7482 raft_consensus.cc:3037] T d558bcaf20bd46548936d5fb946d7710 P 31a72286397c4edc9a63eca3412be168 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:28.730844 7350 raft_consensus.cc:2454] T d558bcaf20bd46548936d5fb946d7710 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:28.731181 7551 leader_election.cc:302] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:28.731436 9071 raft_consensus.cc:2781] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:28.731550 9071 raft_consensus.cc:686] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Becoming Leader. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Running, Role: LEADER
I20240502 21:27:28.731920 7482 raft_consensus.cc:2454] T d558bcaf20bd46548936d5fb946d7710 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:27:28.732100 9071 consensus_queue.cc:237] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:28.736222 4292 catalog_manager.cc:5484] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ddd486109f5434cace13d60d633ac04 (127.29.82.72). New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:27:29.163059 9457 consensus_queue.cc:1035] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:29.178320 9071 consensus_queue.cc:1035] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:29.753090 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:29.781314 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:29.814311 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:29.850591 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:29.864439 4555 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 21:27:29.879385 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:27:29.883886 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:29.900282 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:30.557379 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:30.561796 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:30.565145 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:30.571127 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:30.604084 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:30.675050 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:30.694307 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:27:30.727171 4555 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 21:27:31.362599 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:31.381774 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:31.410110 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:31.422359 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:31.452930 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:31.466367 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:31.497532 4555 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 21:27:31.504046 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 21:27:31.582697 7281 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.68:42449, user_credentials={real_user=slave}} blocked reactor thread for 50736.6us
I20240502 21:27:31.904563 4574 consensus_queue.cc:237] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:31.908062 7616 raft_consensus.cc:1261] T 4c06f3bc97254bff8181d3f6f02470b4 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:31.908615 7213 raft_consensus.cc:1261] T 4c06f3bc97254bff8181d3f6f02470b4 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:31.909054 9320 consensus_queue.cc:1035] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:31.909276 9694 consensus_queue.cc:1035] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:27:31.909682 4508 consensus_peers.cc:485] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c06f3bc97254bff8181d3f6f02470b4. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:31.913043 9320 raft_consensus.cc:2932] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:31.913635 7616 raft_consensus.cc:2932] T 4c06f3bc97254bff8181d3f6f02470b4 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:31.919533 7213 raft_consensus.cc:2932] T 4c06f3bc97254bff8181d3f6f02470b4 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:31.921623 4299 catalog_manager.cc:5484] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:31.971688 4839 consensus_queue.cc:237] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:31.973924 4574 raft_consensus.cc:1261] T 86fa4255c6554ba28e141de3cb328976 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:31.974215 4703 raft_consensus.cc:1261] T 86fa4255c6554ba28e141de3cb328976 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:27:31.974269 4771 consensus_peers.cc:485] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): Couldn't send request to peer 0ddd486109f5434cace13d60d633ac04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86fa4255c6554ba28e141de3cb328976. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:31.974694 9674 consensus_queue.cc:1035] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:31.974964 9522 consensus_queue.cc:1035] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:31.978106 9674 raft_consensus.cc:2932] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:31.978727 4703 raft_consensus.cc:2932] T 86fa4255c6554ba28e141de3cb328976 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:31.989082 4292 catalog_manager.cc:5484] T 86fa4255c6554ba28e141de3cb328976 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:31.993314 4574 raft_consensus.cc:2932] T 86fa4255c6554ba28e141de3cb328976 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:32.061120 4574 consensus_queue.cc:237] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:32.063391 7616 raft_consensus.cc:1261] T 796c7f3cc1f9426daecc89add3c6aa57 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:32.063486 7482 raft_consensus.cc:1261] T 796c7f3cc1f9426daecc89add3c6aa57 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:27:32.063861 4508 consensus_peers.cc:485] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8 -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 796c7f3cc1f9426daecc89add3c6aa57. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:32.064303 9320 consensus_queue.cc:1035] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:32.064584 9697 consensus_queue.cc:1035] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:32.068286 9697 raft_consensus.cc:2932] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:32.068854 7482 raft_consensus.cc:2932] T 796c7f3cc1f9426daecc89add3c6aa57 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:32.073808 7616 raft_consensus.cc:2932] T 796c7f3cc1f9426daecc89add3c6aa57 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:32.076936 4300 catalog_manager.cc:5484] T 796c7f3cc1f9426daecc89add3c6aa57 P 31a72286397c4edc9a63eca3412be168 reported cstate change: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:32.127233 7212 consensus_queue.cc:237] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
W20240502 21:27:32.132987 7151 consensus_peers.cc:485] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 030fa30553294e51808df57d6bb70b66. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:32.133026 4440 raft_consensus.cc:1261] T 030fa30553294e51808df57d6bb70b66 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:32.133867 9068 consensus_queue.cc:1035] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:32.134124 7348 raft_consensus.cc:1261] T 030fa30553294e51808df57d6bb70b66 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:32.134866 9068 consensus_queue.cc:1035] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:32.138127 9500 raft_consensus.cc:2932] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:32.138990 7348 raft_consensus.cc:2932] T 030fa30553294e51808df57d6bb70b66 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:32.147842 4440 raft_consensus.cc:2932] T 030fa30553294e51808df57d6bb70b66 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:32.152045 4299 catalog_manager.cc:5484] T 030fa30553294e51808df57d6bb70b66 P d0456c5e55a542128688904e003d403c reported cstate change: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:32.197685 7616 consensus_queue.cc:237] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:32.200358 4574 raft_consensus.cc:1261] T e8f42cdee5c24db982392d2057b588a2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:27:32.200845 7549 consensus_peers.cc:485] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e8f42cdee5c24db982392d2057b588a2. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:32.201220 9457 consensus_queue.cc:1035] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:32.202039 7212 raft_consensus.cc:1261] T e8f42cdee5c24db982392d2057b588a2 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:32.203032 9457 consensus_queue.cc:1035] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:32.206247 9778 raft_consensus.cc:2932] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:32.206993 7212 raft_consensus.cc:2932] T e8f42cdee5c24db982392d2057b588a2 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:32.210722 4574 raft_consensus.cc:2932] T e8f42cdee5c24db982392d2057b588a2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:32.213382 4299 catalog_manager.cc:5484] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:32.295176 4574 consensus_queue.cc:237] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
W20240502 21:27:32.298646 4509 consensus_peers.cc:485] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 -> Peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): Couldn't send request to peer 0ddd486109f5434cace13d60d633ac04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9fe2e18aa6c407fbbcbb74bf45bd520. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:32.299188 4703 raft_consensus.cc:1261] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:32.300072 9320 consensus_queue.cc:1035] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:32.300626 4839 raft_consensus.cc:1261] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:32.301600 9697 consensus_queue.cc:1035] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:32.306190 9697 raft_consensus.cc:2932] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:32.307094 4839 raft_consensus.cc:2932] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:32.313447 4292 catalog_manager.cc:5484] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:32.314225 4706 raft_consensus.cc:2932] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:32.352790 9802 ts_tablet_manager.cc:902] T 4c06f3bc97254bff8181d3f6f02470b4 P 71e1375b799b439fa3c83ca763c039c1: Initiating tablet copy from peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:27:32.415608 9802 tablet_copy_client.cc:308] T 4c06f3bc97254bff8181d3f6f02470b4 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.66:35781
I20240502 21:27:32.417276 4595 tablet_copy_service.cc:140] P 7a736a8ebb164ce0a051c0fccd76c3b8: Received BeginTabletCopySession request for tablet 4c06f3bc97254bff8181d3f6f02470b4 from peer 71e1375b799b439fa3c83ca763c039c1 ({username='slave'} at 127.29.82.65:57487)
I20240502 21:27:32.417567 4595 tablet_copy_service.cc:161] P 7a736a8ebb164ce0a051c0fccd76c3b8: Beginning new tablet copy session on tablet 4c06f3bc97254bff8181d3f6f02470b4 from peer 71e1375b799b439fa3c83ca763c039c1 at {username='slave'} at 127.29.82.65:57487: session id = 71e1375b799b439fa3c83ca763c039c1-4c06f3bc97254bff8181d3f6f02470b4
I20240502 21:27:32.422704 4595 tablet_copy_source_session.cc:205] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:32.426093 9802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c06f3bc97254bff8181d3f6f02470b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:32.432238 9802 tablet_copy_client.cc:783] T 4c06f3bc97254bff8181d3f6f02470b4 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:32.432680 9802 tablet_copy_client.cc:647] T 4c06f3bc97254bff8181d3f6f02470b4 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:32.435292 9802 tablet_copy_client.cc:523] T 4c06f3bc97254bff8181d3f6f02470b4 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:32.438932 9802 tablet_bootstrap.cc:492] T 4c06f3bc97254bff8181d3f6f02470b4 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:32.446419 9802 tablet_bootstrap.cc:492] T 4c06f3bc97254bff8181d3f6f02470b4 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:32.446763 9802 tablet_bootstrap.cc:492] T 4c06f3bc97254bff8181d3f6f02470b4 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap complete.
I20240502 21:27:32.447000 9802 ts_tablet_manager.cc:1367] T 4c06f3bc97254bff8181d3f6f02470b4 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20240502 21:27:32.447731 9802 raft_consensus.cc:348] T 4c06f3bc97254bff8181d3f6f02470b4 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:32.447930 9802 raft_consensus.cc:729] T 4c06f3bc97254bff8181d3f6f02470b4 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:32.448168 9802 consensus_queue.cc:260] T 4c06f3bc97254bff8181d3f6f02470b4 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:32.448647 9802 ts_tablet_manager.cc:1398] T 4c06f3bc97254bff8181d3f6f02470b4 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:32.449455 4595 tablet_copy_service.cc:342] P 7a736a8ebb164ce0a051c0fccd76c3b8: Request end of tablet copy session 71e1375b799b439fa3c83ca763c039c1-4c06f3bc97254bff8181d3f6f02470b4 received from {username='slave'} at 127.29.82.65:57487
I20240502 21:27:32.449613 4595 tablet_copy_service.cc:433] P 7a736a8ebb164ce0a051c0fccd76c3b8: ending tablet copy session 71e1375b799b439fa3c83ca763c039c1-4c06f3bc97254bff8181d3f6f02470b4 on tablet 4c06f3bc97254bff8181d3f6f02470b4 with peer 71e1375b799b439fa3c83ca763c039c1
I20240502 21:27:32.460392 7617 consensus_queue.cc:237] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:32.463266 7348 raft_consensus.cc:1261] T 616eda3cb218475c8bf0108cc6b90810 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:32.464457 7482 raft_consensus.cc:1261] T 616eda3cb218475c8bf0108cc6b90810 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:32.464589 9071 consensus_queue.cc:1035] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:32.465319 9071 consensus_queue.cc:1035] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:27:32.475960 7549 consensus_peers.cc:485] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 616eda3cb218475c8bf0108cc6b90810. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:32.482721 9457 raft_consensus.cc:2932] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:32.483355 7482 raft_consensus.cc:2932] T 616eda3cb218475c8bf0108cc6b90810 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:32.486172 9827 ts_tablet_manager.cc:902] T 86fa4255c6554ba28e141de3cb328976 P 0ddd486109f5434cace13d60d633ac04: Initiating tablet copy from peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:32.490909 4299 catalog_manager.cc:5484] T 616eda3cb218475c8bf0108cc6b90810 P 31a72286397c4edc9a63eca3412be168 reported cstate change: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:32.493342 7349 raft_consensus.cc:2932] T 616eda3cb218475c8bf0108cc6b90810 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:32.505235 9827 tablet_copy_client.cc:308] T 86fa4255c6554ba28e141de3cb328976 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.68:42449
I20240502 21:27:32.506405 4859 tablet_copy_service.cc:140] P 73a1f2ca1f544de9b0e3b37015f85328: Received BeginTabletCopySession request for tablet 86fa4255c6554ba28e141de3cb328976 from peer 0ddd486109f5434cace13d60d633ac04 ({username='slave'} at 127.29.82.72:48273)
I20240502 21:27:32.506623 4859 tablet_copy_service.cc:161] P 73a1f2ca1f544de9b0e3b37015f85328: Beginning new tablet copy session on tablet 86fa4255c6554ba28e141de3cb328976 from peer 0ddd486109f5434cace13d60d633ac04 at {username='slave'} at 127.29.82.72:48273: session id = 0ddd486109f5434cace13d60d633ac04-86fa4255c6554ba28e141de3cb328976
I20240502 21:27:32.510924 4859 tablet_copy_source_session.cc:205] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:32.520516 9827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86fa4255c6554ba28e141de3cb328976. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:32.526588 9827 tablet_copy_client.cc:783] T 86fa4255c6554ba28e141de3cb328976 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:32.527071 9827 tablet_copy_client.cc:647] T 86fa4255c6554ba28e141de3cb328976 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:32.532013 9827 tablet_copy_client.cc:523] T 86fa4255c6554ba28e141de3cb328976 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:32.535506 9827 tablet_bootstrap.cc:492] T 86fa4255c6554ba28e141de3cb328976 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:32.579011 7349 consensus_queue.cc:237] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:32.585994 7482 raft_consensus.cc:1261] T fed543e42f844227a14a8d812ce158ba P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:27:32.586705 7285 consensus_peers.cc:485] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fed543e42f844227a14a8d812ce158ba. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:32.587350 9676 consensus_queue.cc:1035] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:32.588613 4703 raft_consensus.cc:1261] T fed543e42f844227a14a8d812ce158ba P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:32.590214 9676 consensus_queue.cc:1035] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:32.602901 9827 tablet_bootstrap.cc:492] T 86fa4255c6554ba28e141de3cb328976 P 0ddd486109f5434cace13d60d633ac04: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:32.603554 9827 tablet_bootstrap.cc:492] T 86fa4255c6554ba28e141de3cb328976 P 0ddd486109f5434cace13d60d633ac04: Bootstrap complete.
I20240502 21:27:32.604729 9827 ts_tablet_manager.cc:1367] T 86fa4255c6554ba28e141de3cb328976 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.069s user 0.050s sys 0.013s
I20240502 21:27:32.605628 9827 raft_consensus.cc:348] T 86fa4255c6554ba28e141de3cb328976 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:32.605849 9827 raft_consensus.cc:729] T 86fa4255c6554ba28e141de3cb328976 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:32.606083 9827 consensus_queue.cc:260] T 86fa4255c6554ba28e141de3cb328976 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:32.606774 9676 raft_consensus.cc:2932] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:32.608079 9827 ts_tablet_manager.cc:1398] T 86fa4255c6554ba28e141de3cb328976 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:27:32.610657 4859 tablet_copy_service.cc:342] P 73a1f2ca1f544de9b0e3b37015f85328: Request end of tablet copy session 0ddd486109f5434cace13d60d633ac04-86fa4255c6554ba28e141de3cb328976 received from {username='slave'} at 127.29.82.72:48273
I20240502 21:27:32.610814 4859 tablet_copy_service.cc:433] P 73a1f2ca1f544de9b0e3b37015f85328: ending tablet copy session 0ddd486109f5434cace13d60d633ac04-86fa4255c6554ba28e141de3cb328976 on tablet 86fa4255c6554ba28e141de3cb328976 with peer 0ddd486109f5434cace13d60d633ac04
I20240502 21:27:32.609831 7482 raft_consensus.cc:2932] T fed543e42f844227a14a8d812ce158ba P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:32.620496 4703 raft_consensus.cc:2932] T fed543e42f844227a14a8d812ce158ba P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:32.625543 4299 catalog_manager.cc:5484] T fed543e42f844227a14a8d812ce158ba P 31a72286397c4edc9a63eca3412be168 reported cstate change: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:32.635880 9853 ts_tablet_manager.cc:902] T 030fa30553294e51808df57d6bb70b66 P 7a736a8ebb164ce0a051c0fccd76c3b8: Initiating tablet copy from peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:32.655737 9856 ts_tablet_manager.cc:902] T 796c7f3cc1f9426daecc89add3c6aa57 P 73a1f2ca1f544de9b0e3b37015f85328: Initiating tablet copy from peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:27:32.656111 9853 tablet_copy_client.cc:308] T 030fa30553294e51808df57d6bb70b66 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.69:33193
I20240502 21:27:32.657305 7236 tablet_copy_service.cc:140] P 39981a8dfe94462b9a73ba489af4c714: Received BeginTabletCopySession request for tablet 030fa30553294e51808df57d6bb70b66 from peer 7a736a8ebb164ce0a051c0fccd76c3b8 ({username='slave'} at 127.29.82.66:45181)
I20240502 21:27:32.657553 7236 tablet_copy_service.cc:161] P 39981a8dfe94462b9a73ba489af4c714: Beginning new tablet copy session on tablet 030fa30553294e51808df57d6bb70b66 from peer 7a736a8ebb164ce0a051c0fccd76c3b8 at {username='slave'} at 127.29.82.66:45181: session id = 7a736a8ebb164ce0a051c0fccd76c3b8-030fa30553294e51808df57d6bb70b66
I20240502 21:27:32.661458 7236 tablet_copy_source_session.cc:205] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:32.666167 9853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 030fa30553294e51808df57d6bb70b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:32.666759 9856 tablet_copy_client.cc:308] T 796c7f3cc1f9426daecc89add3c6aa57 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.66:35781
I20240502 21:27:32.667495 4595 tablet_copy_service.cc:140] P 7a736a8ebb164ce0a051c0fccd76c3b8: Received BeginTabletCopySession request for tablet 796c7f3cc1f9426daecc89add3c6aa57 from peer 73a1f2ca1f544de9b0e3b37015f85328 ({username='slave'} at 127.29.82.68:46725)
I20240502 21:27:32.667663 4595 tablet_copy_service.cc:161] P 7a736a8ebb164ce0a051c0fccd76c3b8: Beginning new tablet copy session on tablet 796c7f3cc1f9426daecc89add3c6aa57 from peer 73a1f2ca1f544de9b0e3b37015f85328 at {username='slave'} at 127.29.82.68:46725: session id = 73a1f2ca1f544de9b0e3b37015f85328-796c7f3cc1f9426daecc89add3c6aa57
I20240502 21:27:32.669831 4595 tablet_copy_source_session.cc:205] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:32.671157 9856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 796c7f3cc1f9426daecc89add3c6aa57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:32.675779 9853 tablet_copy_client.cc:783] T 030fa30553294e51808df57d6bb70b66 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:32.676107 9853 tablet_copy_client.cc:647] T 030fa30553294e51808df57d6bb70b66 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:32.677107 9856 tablet_copy_client.cc:783] T 796c7f3cc1f9426daecc89add3c6aa57 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:32.677366 9856 tablet_copy_client.cc:647] T 796c7f3cc1f9426daecc89add3c6aa57 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:32.679793 9856 tablet_copy_client.cc:523] T 796c7f3cc1f9426daecc89add3c6aa57 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:32.683715 9853 tablet_copy_client.cc:523] T 030fa30553294e51808df57d6bb70b66 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:32.687498 9853 tablet_bootstrap.cc:492] T 030fa30553294e51808df57d6bb70b66 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:32.689805 9856 tablet_bootstrap.cc:492] T 796c7f3cc1f9426daecc89add3c6aa57 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:32.696658 9856 tablet_bootstrap.cc:492] T 796c7f3cc1f9426daecc89add3c6aa57 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:32.697063 9856 tablet_bootstrap.cc:492] T 796c7f3cc1f9426daecc89add3c6aa57 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap complete.
I20240502 21:27:32.697311 9856 ts_tablet_manager.cc:1367] T 796c7f3cc1f9426daecc89add3c6aa57 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240502 21:27:32.697780 9853 tablet_bootstrap.cc:492] T 030fa30553294e51808df57d6bb70b66 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:32.698267 9853 tablet_bootstrap.cc:492] T 030fa30553294e51808df57d6bb70b66 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap complete.
I20240502 21:27:32.698089 9856 raft_consensus.cc:348] T 796c7f3cc1f9426daecc89add3c6aa57 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:32.698360 9856 raft_consensus.cc:729] T 796c7f3cc1f9426daecc89add3c6aa57 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:32.698526 9853 ts_tablet_manager.cc:1367] T 030fa30553294e51808df57d6bb70b66 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.001s
I20240502 21:27:32.698622 9856 consensus_queue.cc:260] T 796c7f3cc1f9426daecc89add3c6aa57 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:32.699393 9856 ts_tablet_manager.cc:1398] T 796c7f3cc1f9426daecc89add3c6aa57 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:32.699265 9853 raft_consensus.cc:348] T 030fa30553294e51808df57d6bb70b66 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:32.699477 9853 raft_consensus.cc:729] T 030fa30553294e51808df57d6bb70b66 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:32.699740 9853 consensus_queue.cc:260] T 030fa30553294e51808df57d6bb70b66 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:32.700481 4595 tablet_copy_service.cc:342] P 7a736a8ebb164ce0a051c0fccd76c3b8: Request end of tablet copy session 73a1f2ca1f544de9b0e3b37015f85328-796c7f3cc1f9426daecc89add3c6aa57 received from {username='slave'} at 127.29.82.68:46725
I20240502 21:27:32.700594 4595 tablet_copy_service.cc:433] P 7a736a8ebb164ce0a051c0fccd76c3b8: ending tablet copy session 73a1f2ca1f544de9b0e3b37015f85328-796c7f3cc1f9426daecc89add3c6aa57 on tablet 796c7f3cc1f9426daecc89add3c6aa57 with peer 73a1f2ca1f544de9b0e3b37015f85328
I20240502 21:27:32.701848 9853 ts_tablet_manager.cc:1398] T 030fa30553294e51808df57d6bb70b66 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:32.702637 7236 tablet_copy_service.cc:342] P 39981a8dfe94462b9a73ba489af4c714: Request end of tablet copy session 7a736a8ebb164ce0a051c0fccd76c3b8-030fa30553294e51808df57d6bb70b66 received from {username='slave'} at 127.29.82.66:45181
I20240502 21:27:32.702822 7236 tablet_copy_service.cc:433] P 39981a8dfe94462b9a73ba489af4c714: ending tablet copy session 7a736a8ebb164ce0a051c0fccd76c3b8-030fa30553294e51808df57d6bb70b66 on tablet 030fa30553294e51808df57d6bb70b66 with peer 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:27:32.704478 9802 ts_tablet_manager.cc:902] T e8f42cdee5c24db982392d2057b588a2 P 71e1375b799b439fa3c83ca763c039c1: Initiating tablet copy from peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:32.708933 4839 consensus_queue.cc:237] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:32.716682 7212 raft_consensus.cc:1261] T 202f8b10d5674afa875da2f3929bd8da P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:32.717247 9723 consensus_queue.cc:1035] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:27:32.717903 4770 consensus_peers.cc:485] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 202f8b10d5674afa875da2f3929bd8da. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:32.721258 7616 raft_consensus.cc:1261] T 202f8b10d5674afa875da2f3929bd8da P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:32.721891 9723 consensus_queue.cc:1035] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:32.732267 9802 tablet_copy_client.cc:308] T e8f42cdee5c24db982392d2057b588a2 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.72:41223
I20240502 21:27:32.733109 7638 tablet_copy_service.cc:140] P 0ddd486109f5434cace13d60d633ac04: Received BeginTabletCopySession request for tablet e8f42cdee5c24db982392d2057b588a2 from peer 71e1375b799b439fa3c83ca763c039c1 ({username='slave'} at 127.29.82.65:47155)
I20240502 21:27:32.733314 7638 tablet_copy_service.cc:161] P 0ddd486109f5434cace13d60d633ac04: Beginning new tablet copy session on tablet e8f42cdee5c24db982392d2057b588a2 from peer 71e1375b799b439fa3c83ca763c039c1 at {username='slave'} at 127.29.82.65:47155: session id = 71e1375b799b439fa3c83ca763c039c1-e8f42cdee5c24db982392d2057b588a2
I20240502 21:27:32.737502 7638 tablet_copy_source_session.cc:205] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:32.739180 9802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8f42cdee5c24db982392d2057b588a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:32.745429 9802 tablet_copy_client.cc:783] T e8f42cdee5c24db982392d2057b588a2 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:32.745740 9802 tablet_copy_client.cc:647] T e8f42cdee5c24db982392d2057b588a2 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:32.749683 9723 raft_consensus.cc:2932] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:32.752714 9802 tablet_copy_client.cc:523] T e8f42cdee5c24db982392d2057b588a2 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:32.755185 7212 raft_consensus.cc:2932] T 202f8b10d5674afa875da2f3929bd8da P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:32.755941 9802 tablet_bootstrap.cc:492] T e8f42cdee5c24db982392d2057b588a2 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:32.759593 4292 catalog_manager.cc:5484] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:32.755548 7616 raft_consensus.cc:2932] T 202f8b10d5674afa875da2f3929bd8da P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:32.789414 9827 ts_tablet_manager.cc:902] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 0ddd486109f5434cace13d60d633ac04: Initiating tablet copy from peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:27:32.799901 9802 tablet_bootstrap.cc:492] T e8f42cdee5c24db982392d2057b588a2 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:32.806766 9802 tablet_bootstrap.cc:492] T e8f42cdee5c24db982392d2057b588a2 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap complete.
I20240502 21:27:32.807062 9802 ts_tablet_manager.cc:1367] T e8f42cdee5c24db982392d2057b588a2 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.051s user 0.015s sys 0.019s
I20240502 21:27:32.808760 9802 raft_consensus.cc:348] T e8f42cdee5c24db982392d2057b588a2 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:32.809070 9802 raft_consensus.cc:729] T e8f42cdee5c24db982392d2057b588a2 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:32.811378 9802 consensus_queue.cc:260] T e8f42cdee5c24db982392d2057b588a2 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:32.813633 9802 ts_tablet_manager.cc:1398] T e8f42cdee5c24db982392d2057b588a2 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 21:27:32.814327 7638 tablet_copy_service.cc:342] P 0ddd486109f5434cace13d60d633ac04: Request end of tablet copy session 71e1375b799b439fa3c83ca763c039c1-e8f42cdee5c24db982392d2057b588a2 received from {username='slave'} at 127.29.82.65:47155
I20240502 21:27:32.814556 7638 tablet_copy_service.cc:433] P 0ddd486109f5434cace13d60d633ac04: ending tablet copy session 71e1375b799b439fa3c83ca763c039c1-e8f42cdee5c24db982392d2057b588a2 on tablet e8f42cdee5c24db982392d2057b588a2 with peer 71e1375b799b439fa3c83ca763c039c1
I20240502 21:27:32.864939 9827 tablet_copy_client.cc:308] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.66:35781
I20240502 21:27:32.866005 4595 tablet_copy_service.cc:140] P 7a736a8ebb164ce0a051c0fccd76c3b8: Received BeginTabletCopySession request for tablet c9fe2e18aa6c407fbbcbb74bf45bd520 from peer 0ddd486109f5434cace13d60d633ac04 ({username='slave'} at 127.29.82.72:40279)
I20240502 21:27:32.866169 4595 tablet_copy_service.cc:161] P 7a736a8ebb164ce0a051c0fccd76c3b8: Beginning new tablet copy session on tablet c9fe2e18aa6c407fbbcbb74bf45bd520 from peer 0ddd486109f5434cace13d60d633ac04 at {username='slave'} at 127.29.82.72:40279: session id = 0ddd486109f5434cace13d60d633ac04-c9fe2e18aa6c407fbbcbb74bf45bd520
I20240502 21:27:32.873356 4595 tablet_copy_source_session.cc:205] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:32.874823 9827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9fe2e18aa6c407fbbcbb74bf45bd520. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:32.883423 9827 tablet_copy_client.cc:783] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:32.884802 9827 tablet_copy_client.cc:647] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:32.888562 9827 tablet_copy_client.cc:523] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:32.892407 9827 tablet_bootstrap.cc:492] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:32.930024 4440 raft_consensus.cc:1203] T 4c06f3bc97254bff8181d3f6f02470b4 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:32.946290 4440 consensus_queue.cc:237] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:32.949199 4706 raft_consensus.cc:1261] T a7a0167853fa420586455b82fb893c46 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:27:32.949276 4376 consensus_peers.cc:485] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 -> Peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): Couldn't send request to peer 0ddd486109f5434cace13d60d633ac04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7a0167853fa420586455b82fb893c46. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:32.949645 4839 raft_consensus.cc:1261] T a7a0167853fa420586455b82fb893c46 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:32.949884 9913 consensus_queue.cc:1035] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:32.950201 9809 consensus_queue.cc:1035] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:32.953105 9809 raft_consensus.cc:2932] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:32.954898 4706 raft_consensus.cc:2932] T a7a0167853fa420586455b82fb893c46 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:32.957885 4839 raft_consensus.cc:2932] T a7a0167853fa420586455b82fb893c46 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:32.960963 9827 tablet_bootstrap.cc:492] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 0ddd486109f5434cace13d60d633ac04: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:32.961407 9827 tablet_bootstrap.cc:492] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 0ddd486109f5434cace13d60d633ac04: Bootstrap complete.
I20240502 21:27:32.961684 9827 ts_tablet_manager.cc:1367] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.069s user 0.058s sys 0.008s
I20240502 21:27:32.962070 4292 catalog_manager.cc:5484] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:32.962563 9827 raft_consensus.cc:348] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:32.962796 9827 raft_consensus.cc:729] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:32.963043 9827 consensus_queue.cc:260] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:32.963706 9827 ts_tablet_manager.cc:1398] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20240502 21:27:32.966403 4595 tablet_copy_service.cc:342] P 7a736a8ebb164ce0a051c0fccd76c3b8: Request end of tablet copy session 0ddd486109f5434cace13d60d633ac04-c9fe2e18aa6c407fbbcbb74bf45bd520 received from {username='slave'} at 127.29.82.72:40279
I20240502 21:27:32.966531 4595 tablet_copy_service.cc:433] P 7a736a8ebb164ce0a051c0fccd76c3b8: ending tablet copy session 0ddd486109f5434cace13d60d633ac04-c9fe2e18aa6c407fbbcbb74bf45bd520 on tablet c9fe2e18aa6c407fbbcbb74bf45bd520 with peer 0ddd486109f5434cace13d60d633ac04
I20240502 21:27:32.982920 9853 ts_tablet_manager.cc:902] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8: Initiating tablet copy from peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:32.984215 9853 tablet_copy_client.cc:308] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.70:40719
I20240502 21:27:32.985241 7370 tablet_copy_service.cc:140] P d0456c5e55a542128688904e003d403c: Received BeginTabletCopySession request for tablet fed543e42f844227a14a8d812ce158ba from peer 7a736a8ebb164ce0a051c0fccd76c3b8 ({username='slave'} at 127.29.82.66:56529)
I20240502 21:27:32.985486 7370 tablet_copy_service.cc:161] P d0456c5e55a542128688904e003d403c: Beginning new tablet copy session on tablet fed543e42f844227a14a8d812ce158ba from peer 7a736a8ebb164ce0a051c0fccd76c3b8 at {username='slave'} at 127.29.82.66:56529: session id = 7a736a8ebb164ce0a051c0fccd76c3b8-fed543e42f844227a14a8d812ce158ba
I20240502 21:27:32.991979 7370 tablet_copy_source_session.cc:205] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:32.994503 9853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fed543e42f844227a14a8d812ce158ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:32.999205 9853 tablet_copy_client.cc:783] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:32.999395 9853 tablet_copy_client.cc:647] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:33.003922 9856 ts_tablet_manager.cc:902] T 616eda3cb218475c8bf0108cc6b90810 P 73a1f2ca1f544de9b0e3b37015f85328: Initiating tablet copy from peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:33.005949 9853 tablet_copy_client.cc:523] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:33.011803 9853 tablet_bootstrap.cc:492] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:33.012383 9856 tablet_copy_client.cc:308] T 616eda3cb218475c8bf0108cc6b90810 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.72:41223
I20240502 21:27:33.013317 7638 tablet_copy_service.cc:140] P 0ddd486109f5434cace13d60d633ac04: Received BeginTabletCopySession request for tablet 616eda3cb218475c8bf0108cc6b90810 from peer 73a1f2ca1f544de9b0e3b37015f85328 ({username='slave'} at 127.29.82.68:47485)
I20240502 21:27:33.013515 7638 tablet_copy_service.cc:161] P 0ddd486109f5434cace13d60d633ac04: Beginning new tablet copy session on tablet 616eda3cb218475c8bf0108cc6b90810 from peer 73a1f2ca1f544de9b0e3b37015f85328 at {username='slave'} at 127.29.82.68:47485: session id = 73a1f2ca1f544de9b0e3b37015f85328-616eda3cb218475c8bf0108cc6b90810
I20240502 21:27:33.016716 7638 tablet_copy_source_session.cc:205] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:33.018406 9856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 616eda3cb218475c8bf0108cc6b90810. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:33.026150 9856 tablet_copy_client.cc:783] T 616eda3cb218475c8bf0108cc6b90810 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:33.026473 9856 tablet_copy_client.cc:647] T 616eda3cb218475c8bf0108cc6b90810 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:33.030673 9856 tablet_copy_client.cc:523] T 616eda3cb218475c8bf0108cc6b90810 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:33.036078 9856 tablet_bootstrap.cc:492] T 616eda3cb218475c8bf0108cc6b90810 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:33.040988 7616 raft_consensus.cc:1203] T 86fa4255c6554ba28e141de3cb328976 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:33.054919 9853 tablet_bootstrap.cc:492] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:33.055297 9856 tablet_bootstrap.cc:492] T 616eda3cb218475c8bf0108cc6b90810 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:33.055545 9853 tablet_bootstrap.cc:492] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap complete.
I20240502 21:27:33.055843 9856 tablet_bootstrap.cc:492] T 616eda3cb218475c8bf0108cc6b90810 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap complete.
I20240502 21:27:33.056103 9853 ts_tablet_manager.cc:1367] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.044s user 0.017s sys 0.010s
I20240502 21:27:33.056103 9856 ts_tablet_manager.cc:1367] T 616eda3cb218475c8bf0108cc6b90810 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.010s
I20240502 21:27:33.056890 9853 raft_consensus.cc:348] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:33.056879 9856 raft_consensus.cc:348] T 616eda3cb218475c8bf0108cc6b90810 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:33.057164 9856 raft_consensus.cc:729] T 616eda3cb218475c8bf0108cc6b90810 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:33.057287 9853 raft_consensus.cc:729] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:33.057561 9853 consensus_queue.cc:260] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:33.057485 9856 consensus_queue.cc:260] T 616eda3cb218475c8bf0108cc6b90810 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:33.058828 9856 ts_tablet_manager.cc:1398] T 616eda3cb218475c8bf0108cc6b90810 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:33.059366 9853 ts_tablet_manager.cc:1398] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:33.060784 7638 tablet_copy_service.cc:342] P 0ddd486109f5434cace13d60d633ac04: Request end of tablet copy session 73a1f2ca1f544de9b0e3b37015f85328-616eda3cb218475c8bf0108cc6b90810 received from {username='slave'} at 127.29.82.68:47485
I20240502 21:27:33.060937 7638 tablet_copy_service.cc:433] P 0ddd486109f5434cace13d60d633ac04: ending tablet copy session 73a1f2ca1f544de9b0e3b37015f85328-616eda3cb218475c8bf0108cc6b90810 on tablet 616eda3cb218475c8bf0108cc6b90810 with peer 73a1f2ca1f544de9b0e3b37015f85328
I20240502 21:27:33.061203 7370 tablet_copy_service.cc:342] P d0456c5e55a542128688904e003d403c: Request end of tablet copy session 7a736a8ebb164ce0a051c0fccd76c3b8-fed543e42f844227a14a8d812ce158ba received from {username='slave'} at 127.29.82.66:56529
I20240502 21:27:33.061409 7370 tablet_copy_service.cc:433] P d0456c5e55a542128688904e003d403c: ending tablet copy session 7a736a8ebb164ce0a051c0fccd76c3b8-fed543e42f844227a14a8d812ce158ba on tablet fed543e42f844227a14a8d812ce158ba with peer 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:27:33.084350 7213 consensus_queue.cc:237] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:33.087214 7483 raft_consensus.cc:1261] T 4cc82d5106804490993007f9f75943aa P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:33.087548 7616 raft_consensus.cc:1261] T 4cc82d5106804490993007f9f75943aa P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:33.088079 9500 consensus_queue.cc:1035] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:27:33.088483 7148 consensus_peers.cc:485] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4cc82d5106804490993007f9f75943aa. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:33.088512 9760 consensus_queue.cc:1035] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:33.091992 4574 raft_consensus.cc:1203] T 030fa30553294e51808df57d6bb70b66 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:33.091858 9760 raft_consensus.cc:2932] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:33.095192 7483 raft_consensus.cc:2932] T 4cc82d5106804490993007f9f75943aa P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:33.102128 7616 raft_consensus.cc:2932] T 4cc82d5106804490993007f9f75943aa P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:33.111205 4292 catalog_manager.cc:5484] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:33.118834 4839 raft_consensus.cc:1203] T 796c7f3cc1f9426daecc89add3c6aa57 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:33.154541 7617 consensus_queue.cc:237] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:33.157414 4440 raft_consensus.cc:1261] T 37d864bed6ef46e4a3f57de0373bbe57 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:33.157452 7348 raft_consensus.cc:1261] T 37d864bed6ef46e4a3f57de0373bbe57 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:33.157979 9457 consensus_queue.cc:1035] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:33.158304 9071 consensus_queue.cc:1035] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:27:33.159617 7552 consensus_peers.cc:485] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37d864bed6ef46e4a3f57de0373bbe57. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:33.166146 9457 raft_consensus.cc:2932] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:33.166756 7348 raft_consensus.cc:2932] T 37d864bed6ef46e4a3f57de0373bbe57 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:33.170933 4442 raft_consensus.cc:2932] T 37d864bed6ef46e4a3f57de0373bbe57 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:33.175370 4292 catalog_manager.cc:5484] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:33.215458 9802 ts_tablet_manager.cc:902] T 202f8b10d5674afa875da2f3929bd8da P 71e1375b799b439fa3c83ca763c039c1: Initiating tablet copy from peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:33.222072 9802 tablet_copy_client.cc:308] T 202f8b10d5674afa875da2f3929bd8da P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.68:42449
I20240502 21:27:33.222765 4859 tablet_copy_service.cc:140] P 73a1f2ca1f544de9b0e3b37015f85328: Received BeginTabletCopySession request for tablet 202f8b10d5674afa875da2f3929bd8da from peer 71e1375b799b439fa3c83ca763c039c1 ({username='slave'} at 127.29.82.65:44725)
I20240502 21:27:33.222899 4859 tablet_copy_service.cc:161] P 73a1f2ca1f544de9b0e3b37015f85328: Beginning new tablet copy session on tablet 202f8b10d5674afa875da2f3929bd8da from peer 71e1375b799b439fa3c83ca763c039c1 at {username='slave'} at 127.29.82.65:44725: session id = 71e1375b799b439fa3c83ca763c039c1-202f8b10d5674afa875da2f3929bd8da
I20240502 21:27:33.223083 7483 consensus_queue.cc:237] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:33.225158 4859 tablet_copy_source_session.cc:205] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:33.226362 4442 raft_consensus.cc:1261] T 402e76e6f92744fbb36653d04f861e4f P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:33.226823 9802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 202f8b10d5674afa875da2f3929bd8da. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 21:27:33.226837 7419 consensus_peers.cc:485] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 -> Peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Couldn't send request to peer 39981a8dfe94462b9a73ba489af4c714. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 402e76e6f92744fbb36653d04f861e4f. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:33.226873 4574 raft_consensus.cc:1261] T 402e76e6f92744fbb36653d04f861e4f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:33.227521 9367 consensus_queue.cc:1035] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:33.229146 9672 consensus_queue.cc:1035] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:33.236477 9802 tablet_copy_client.cc:783] T 202f8b10d5674afa875da2f3929bd8da P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:33.236713 9802 tablet_copy_client.cc:647] T 202f8b10d5674afa875da2f3929bd8da P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:33.237430 9367 raft_consensus.cc:2932] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:33.238433 9802 tablet_copy_client.cc:523] T 202f8b10d5674afa875da2f3929bd8da P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:33.242574 9802 tablet_bootstrap.cc:492] T 202f8b10d5674afa875da2f3929bd8da P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:33.246201 4442 raft_consensus.cc:2932] T 402e76e6f92744fbb36653d04f861e4f P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:33.256160 4300 catalog_manager.cc:5484] T 402e76e6f92744fbb36653d04f861e4f P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: config changed from index -1 to 2, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:33.260321 4575 raft_consensus.cc:2932] T 402e76e6f92744fbb36653d04f861e4f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:33.286104 9802 tablet_bootstrap.cc:492] T 202f8b10d5674afa875da2f3929bd8da P 71e1375b799b439fa3c83ca763c039c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:33.286509 9802 tablet_bootstrap.cc:492] T 202f8b10d5674afa875da2f3929bd8da P 71e1375b799b439fa3c83ca763c039c1: Bootstrap complete.
I20240502 21:27:33.286782 9802 ts_tablet_manager.cc:1367] T 202f8b10d5674afa875da2f3929bd8da P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.044s user 0.013s sys 0.018s
I20240502 21:27:33.287868 9802 raft_consensus.cc:348] T 202f8b10d5674afa875da2f3929bd8da P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:33.288192 9802 raft_consensus.cc:729] T 202f8b10d5674afa875da2f3929bd8da P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:33.288507 9802 consensus_queue.cc:260] T 202f8b10d5674afa875da2f3929bd8da P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:33.289376 9802 ts_tablet_manager.cc:1398] T 202f8b10d5674afa875da2f3929bd8da P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240502 21:27:33.291265 4859 tablet_copy_service.cc:342] P 73a1f2ca1f544de9b0e3b37015f85328: Request end of tablet copy session 71e1375b799b439fa3c83ca763c039c1-202f8b10d5674afa875da2f3929bd8da received from {username='slave'} at 127.29.82.65:44725
I20240502 21:27:33.291380 4859 tablet_copy_service.cc:433] P 73a1f2ca1f544de9b0e3b37015f85328: ending tablet copy session 71e1375b799b439fa3c83ca763c039c1-202f8b10d5674afa875da2f3929bd8da on tablet 202f8b10d5674afa875da2f3929bd8da with peer 71e1375b799b439fa3c83ca763c039c1
I20240502 21:27:33.312724 4442 raft_consensus.cc:1203] T e8f42cdee5c24db982392d2057b588a2 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:33.320991 7617 raft_consensus.cc:1203] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:33.343675 7482 consensus_queue.cc:237] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:33.348910 7617 raft_consensus.cc:1261] T 80304ab48274408393c2e245477c3a78 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:27:33.349663 7420 consensus_peers.cc:485] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80304ab48274408393c2e245477c3a78. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:33.350904 7348 raft_consensus.cc:1261] T 80304ab48274408393c2e245477c3a78 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:33.352221 9672 consensus_queue.cc:1035] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:33.352679 10006 consensus_queue.cc:1035] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:33.369773 9672 raft_consensus.cc:2932] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:33.370441 7348 raft_consensus.cc:2932] T 80304ab48274408393c2e245477c3a78 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:33.377537 4300 catalog_manager.cc:5484] T 80304ab48274408393c2e245477c3a78 P d0456c5e55a542128688904e003d403c reported cstate change: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:33.381654 7617 raft_consensus.cc:2932] T 80304ab48274408393c2e245477c3a78 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:33.423079 4839 raft_consensus.cc:1203] T 616eda3cb218475c8bf0108cc6b90810 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:33.428805 4575 raft_consensus.cc:1203] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:33.436952 4575 consensus_queue.cc:237] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:33.439628 7348 raft_consensus.cc:1261] T d6370cfa3d804e0fa5dc2a0f71d78840 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:33.439823 7482 raft_consensus.cc:1261] T d6370cfa3d804e0fa5dc2a0f71d78840 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:33.440518 9694 consensus_queue.cc:1035] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:33.441190 9694 consensus_queue.cc:1035] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:27:33.451328 4508 consensus_peers.cc:485] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8 -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d6370cfa3d804e0fa5dc2a0f71d78840. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:33.451932 9694 raft_consensus.cc:2932] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:33.453040 7482 raft_consensus.cc:2932] T d6370cfa3d804e0fa5dc2a0f71d78840 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:33.459120 4300 catalog_manager.cc:5484] T d6370cfa3d804e0fa5dc2a0f71d78840 P 31a72286397c4edc9a63eca3412be168 reported cstate change: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:33.462656 7349 raft_consensus.cc:2932] T d6370cfa3d804e0fa5dc2a0f71d78840 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:33.524113 4442 consensus_queue.cc:237] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:33.527125 4575 raft_consensus.cc:1261] T 1c243b27429043aa8642d9b2ed595ab3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:33.527717 4706 raft_consensus.cc:1261] T 1c243b27429043aa8642d9b2ed595ab3 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:33.528450 9809 consensus_queue.cc:1035] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:27:33.528717 4377 consensus_peers.cc:485] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 -> Peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Couldn't send request to peer d0456c5e55a542128688904e003d403c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c243b27429043aa8642d9b2ed595ab3. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:33.528957 9809 consensus_queue.cc:1035] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:33.536945 9913 raft_consensus.cc:2932] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:33.537607 4575 raft_consensus.cc:2932] T 1c243b27429043aa8642d9b2ed595ab3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:33.542667 4706 raft_consensus.cc:2932] T 1c243b27429043aa8642d9b2ed595ab3 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:33.548390 4292 catalog_manager.cc:5484] T 1c243b27429043aa8642d9b2ed595ab3 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: config changed from index -1 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:33.562178 10057 ts_tablet_manager.cc:902] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04: Initiating tablet copy from peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:33.563666 10057 tablet_copy_client.cc:308] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.65:42781
I20240502 21:27:33.564558 4463 tablet_copy_service.cc:140] P 71e1375b799b439fa3c83ca763c039c1: Received BeginTabletCopySession request for tablet a7a0167853fa420586455b82fb893c46 from peer 0ddd486109f5434cace13d60d633ac04 ({username='slave'} at 127.29.82.72:46359)
I20240502 21:27:33.564725 4463 tablet_copy_service.cc:161] P 71e1375b799b439fa3c83ca763c039c1: Beginning new tablet copy session on tablet a7a0167853fa420586455b82fb893c46 from peer 0ddd486109f5434cace13d60d633ac04 at {username='slave'} at 127.29.82.72:46359: session id = 0ddd486109f5434cace13d60d633ac04-a7a0167853fa420586455b82fb893c46
I20240502 21:27:33.567535 4463 tablet_copy_source_session.cc:205] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:33.569429 10057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7a0167853fa420586455b82fb893c46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:33.575356 10057 tablet_copy_client.cc:783] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:33.575721 10057 tablet_copy_client.cc:647] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:33.577369 10057 tablet_copy_client.cc:523] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:33.580539 10057 tablet_bootstrap.cc:492] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:33.615028 4703 consensus_queue.cc:237] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:33.617780 4575 raft_consensus.cc:1261] T 2285212d78b0460ca329949eb3140e69 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:33.618119 4442 raft_consensus.cc:1261] T 2285212d78b0460ca329949eb3140e69 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:33.618706 10075 consensus_queue.cc:1035] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:33.619187 9791 consensus_queue.cc:1035] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:27:33.619515 4640 consensus_peers.cc:485] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb -> Peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): Couldn't send request to peer 0ddd486109f5434cace13d60d633ac04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2285212d78b0460ca329949eb3140e69. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:33.633349 10057 tablet_bootstrap.cc:492] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:33.633888 10057 tablet_bootstrap.cc:492] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04: Bootstrap complete.
I20240502 21:27:33.634181 10057 ts_tablet_manager.cc:1367] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.054s user 0.033s sys 0.016s
I20240502 21:27:33.634953 10057 raft_consensus.cc:348] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:33.635190 10057 raft_consensus.cc:729] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:33.635442 10057 consensus_queue.cc:260] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:33.637023 10057 ts_tablet_manager.cc:1398] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240502 21:27:33.637689 4463 tablet_copy_service.cc:342] P 71e1375b799b439fa3c83ca763c039c1: Request end of tablet copy session 0ddd486109f5434cace13d60d633ac04-a7a0167853fa420586455b82fb893c46 received from {username='slave'} at 127.29.82.72:46359
I20240502 21:27:33.637835 4463 tablet_copy_service.cc:433] P 71e1375b799b439fa3c83ca763c039c1: ending tablet copy session 0ddd486109f5434cace13d60d633ac04-a7a0167853fa420586455b82fb893c46 on tablet a7a0167853fa420586455b82fb893c46 with peer 0ddd486109f5434cace13d60d633ac04
I20240502 21:27:33.638765 10075 raft_consensus.cc:2932] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:33.639705 4575 raft_consensus.cc:2932] T 2285212d78b0460ca329949eb3140e69 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:33.645895 4442 raft_consensus.cc:2932] T 2285212d78b0460ca329949eb3140e69 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:33.648626 4292 catalog_manager.cc:5484] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:33.651754 4440 raft_consensus.cc:1203] T 202f8b10d5674afa875da2f3929bd8da P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:33.686151 4442 consensus_queue.cc:237] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:33.688704 7212 raft_consensus.cc:1261] T 81db5e77d00446cba3eb8345ee3df476 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:33.689123 7482 raft_consensus.cc:1261] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:33.689383 10050 consensus_queue.cc:1035] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:33.689939 9913 consensus_queue.cc:1035] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:27:33.690184 4378 consensus_peers.cc:485] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81db5e77d00446cba3eb8345ee3df476. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:33.693382 10050 raft_consensus.cc:2932] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:33.694218 7484 raft_consensus.cc:2932] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:33.699448 10096 ts_tablet_manager.cc:902] T 4cc82d5106804490993007f9f75943aa P 73a1f2ca1f544de9b0e3b37015f85328: Initiating tablet copy from peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:33.700968 4292 catalog_manager.cc:5484] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:33.709079 10100 ts_tablet_manager.cc:902] T 37d864bed6ef46e4a3f57de0373bbe57 P 7a736a8ebb164ce0a051c0fccd76c3b8: Initiating tablet copy from peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:33.728612 7212 raft_consensus.cc:2932] T 81db5e77d00446cba3eb8345ee3df476 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:33.741575 10104 ts_tablet_manager.cc:902] T 402e76e6f92744fbb36653d04f861e4f P 39981a8dfe94462b9a73ba489af4c714: Initiating tablet copy from peer 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:33.759685 10109 ts_tablet_manager.cc:902] T 80304ab48274408393c2e245477c3a78 P 398c3c8f3195492c900439ad5e1931bb: Initiating tablet copy from peer 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:33.765889 10100 tablet_copy_client.cc:308] T 37d864bed6ef46e4a3f57de0373bbe57 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.72:41223
I20240502 21:27:33.766592 7638 tablet_copy_service.cc:140] P 0ddd486109f5434cace13d60d633ac04: Received BeginTabletCopySession request for tablet 37d864bed6ef46e4a3f57de0373bbe57 from peer 7a736a8ebb164ce0a051c0fccd76c3b8 ({username='slave'} at 127.29.82.66:56819)
I20240502 21:27:33.766767 7638 tablet_copy_service.cc:161] P 0ddd486109f5434cace13d60d633ac04: Beginning new tablet copy session on tablet 37d864bed6ef46e4a3f57de0373bbe57 from peer 7a736a8ebb164ce0a051c0fccd76c3b8 at {username='slave'} at 127.29.82.66:56819: session id = 7a736a8ebb164ce0a051c0fccd76c3b8-37d864bed6ef46e4a3f57de0373bbe57
I20240502 21:27:33.769541 7638 tablet_copy_source_session.cc:205] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:33.773485 10100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37d864bed6ef46e4a3f57de0373bbe57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:33.790239 10100 tablet_copy_client.cc:783] T 37d864bed6ef46e4a3f57de0373bbe57 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:33.790805 10100 tablet_copy_client.cc:647] T 37d864bed6ef46e4a3f57de0373bbe57 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:33.793128 10100 tablet_copy_client.cc:523] T 37d864bed6ef46e4a3f57de0373bbe57 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:33.796393 10100 tablet_bootstrap.cc:492] T 37d864bed6ef46e4a3f57de0373bbe57 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:33.812578 10096 tablet_copy_client.cc:308] T 4cc82d5106804490993007f9f75943aa P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.69:33193
I20240502 21:27:33.813318 7236 tablet_copy_service.cc:140] P 39981a8dfe94462b9a73ba489af4c714: Received BeginTabletCopySession request for tablet 4cc82d5106804490993007f9f75943aa from peer 73a1f2ca1f544de9b0e3b37015f85328 ({username='slave'} at 127.29.82.68:40041)
I20240502 21:27:33.813522 7236 tablet_copy_service.cc:161] P 39981a8dfe94462b9a73ba489af4c714: Beginning new tablet copy session on tablet 4cc82d5106804490993007f9f75943aa from peer 73a1f2ca1f544de9b0e3b37015f85328 at {username='slave'} at 127.29.82.68:40041: session id = 73a1f2ca1f544de9b0e3b37015f85328-4cc82d5106804490993007f9f75943aa
I20240502 21:27:33.816965 7236 tablet_copy_source_session.cc:205] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:33.818930 10096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc82d5106804490993007f9f75943aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:33.822260 10109 tablet_copy_client.cc:308] T 80304ab48274408393c2e245477c3a78 P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.71:33913
I20240502 21:27:33.823521 7505 tablet_copy_service.cc:140] P 31a72286397c4edc9a63eca3412be168: Received BeginTabletCopySession request for tablet 80304ab48274408393c2e245477c3a78 from peer 398c3c8f3195492c900439ad5e1931bb ({username='slave'} at 127.29.82.67:43117)
I20240502 21:27:33.823791 7505 tablet_copy_service.cc:161] P 31a72286397c4edc9a63eca3412be168: Beginning new tablet copy session on tablet 80304ab48274408393c2e245477c3a78 from peer 398c3c8f3195492c900439ad5e1931bb at {username='slave'} at 127.29.82.67:43117: session id = 398c3c8f3195492c900439ad5e1931bb-80304ab48274408393c2e245477c3a78
I20240502 21:27:33.829349 7505 tablet_copy_source_session.cc:205] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:33.830513 10096 tablet_copy_client.cc:783] T 4cc82d5106804490993007f9f75943aa P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:33.830731 10096 tablet_copy_client.cc:647] T 4cc82d5106804490993007f9f75943aa P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:33.832126 10109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80304ab48274408393c2e245477c3a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:33.832989 10096 tablet_copy_client.cc:523] T 4cc82d5106804490993007f9f75943aa P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:33.843143 10104 tablet_copy_client.cc:308] T 402e76e6f92744fbb36653d04f861e4f P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.71:33913
I20240502 21:27:33.850351 10100 tablet_bootstrap.cc:492] T 37d864bed6ef46e4a3f57de0373bbe57 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:33.850819 10100 tablet_bootstrap.cc:492] T 37d864bed6ef46e4a3f57de0373bbe57 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap complete.
I20240502 21:27:33.851166 10100 ts_tablet_manager.cc:1367] T 37d864bed6ef46e4a3f57de0373bbe57 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.002s
I20240502 21:27:33.852216 7505 tablet_copy_service.cc:140] P 31a72286397c4edc9a63eca3412be168: Received BeginTabletCopySession request for tablet 402e76e6f92744fbb36653d04f861e4f from peer 39981a8dfe94462b9a73ba489af4c714 ({username='slave'} at 127.29.82.69:38761)
I20240502 21:27:33.852459 7505 tablet_copy_service.cc:161] P 31a72286397c4edc9a63eca3412be168: Beginning new tablet copy session on tablet 402e76e6f92744fbb36653d04f861e4f from peer 39981a8dfe94462b9a73ba489af4c714 at {username='slave'} at 127.29.82.69:38761: session id = 39981a8dfe94462b9a73ba489af4c714-402e76e6f92744fbb36653d04f861e4f
I20240502 21:27:33.856118 10096 tablet_bootstrap.cc:492] T 4cc82d5106804490993007f9f75943aa P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:33.851984 10100 raft_consensus.cc:348] T 37d864bed6ef46e4a3f57de0373bbe57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:33.856838 10100 raft_consensus.cc:729] T 37d864bed6ef46e4a3f57de0373bbe57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:33.857085 10100 consensus_queue.cc:260] T 37d864bed6ef46e4a3f57de0373bbe57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:33.858215 10100 ts_tablet_manager.cc:1398] T 37d864bed6ef46e4a3f57de0373bbe57 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20240502 21:27:33.863541 7638 tablet_copy_service.cc:342] P 0ddd486109f5434cace13d60d633ac04: Request end of tablet copy session 7a736a8ebb164ce0a051c0fccd76c3b8-37d864bed6ef46e4a3f57de0373bbe57 received from {username='slave'} at 127.29.82.66:56819
I20240502 21:27:33.863909 7638 tablet_copy_service.cc:433] P 0ddd486109f5434cace13d60d633ac04: ending tablet copy session 7a736a8ebb164ce0a051c0fccd76c3b8-37d864bed6ef46e4a3f57de0373bbe57 on tablet 37d864bed6ef46e4a3f57de0373bbe57 with peer 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:27:33.866097 10109 tablet_copy_client.cc:783] T 80304ab48274408393c2e245477c3a78 P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:33.869647 7505 tablet_copy_source_session.cc:205] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:33.870029 10109 tablet_copy_client.cc:647] T 80304ab48274408393c2e245477c3a78 P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:33.871954 10104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 402e76e6f92744fbb36653d04f861e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:33.879616 10104 tablet_copy_client.cc:783] T 402e76e6f92744fbb36653d04f861e4f P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:33.880033 10104 tablet_copy_client.cc:647] T 402e76e6f92744fbb36653d04f861e4f P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:33.883546 10104 tablet_copy_client.cc:523] T 402e76e6f92744fbb36653d04f861e4f P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:33.885128 10109 tablet_copy_client.cc:523] T 80304ab48274408393c2e245477c3a78 P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:33.890515 10109 tablet_bootstrap.cc:492] T 80304ab48274408393c2e245477c3a78 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:33.892663 10104 tablet_bootstrap.cc:492] T 402e76e6f92744fbb36653d04f861e4f P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:33.925164 10096 tablet_bootstrap.cc:492] T 4cc82d5106804490993007f9f75943aa P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:33.925725 10096 tablet_bootstrap.cc:492] T 4cc82d5106804490993007f9f75943aa P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap complete.
I20240502 21:27:33.926043 10096 ts_tablet_manager.cc:1367] T 4cc82d5106804490993007f9f75943aa P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.070s user 0.016s sys 0.003s
I20240502 21:27:33.927001 10096 raft_consensus.cc:348] T 4cc82d5106804490993007f9f75943aa P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:33.927307 10096 raft_consensus.cc:729] T 4cc82d5106804490993007f9f75943aa P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:33.927600 10096 consensus_queue.cc:260] T 4cc82d5106804490993007f9f75943aa P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:33.928109 10109 tablet_bootstrap.cc:492] T 80304ab48274408393c2e245477c3a78 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:33.928594 10109 tablet_bootstrap.cc:492] T 80304ab48274408393c2e245477c3a78 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap complete.
I20240502 21:27:33.930465 10096 ts_tablet_manager.cc:1398] T 4cc82d5106804490993007f9f75943aa P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:27:33.934790 10104 tablet_bootstrap.cc:492] T 402e76e6f92744fbb36653d04f861e4f P 39981a8dfe94462b9a73ba489af4c714: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:33.935484 7236 tablet_copy_service.cc:342] P 39981a8dfe94462b9a73ba489af4c714: Request end of tablet copy session 73a1f2ca1f544de9b0e3b37015f85328-4cc82d5106804490993007f9f75943aa received from {username='slave'} at 127.29.82.68:40041
I20240502 21:27:33.935669 7236 tablet_copy_service.cc:433] P 39981a8dfe94462b9a73ba489af4c714: ending tablet copy session 73a1f2ca1f544de9b0e3b37015f85328-4cc82d5106804490993007f9f75943aa on tablet 4cc82d5106804490993007f9f75943aa with peer 73a1f2ca1f544de9b0e3b37015f85328
I20240502 21:27:33.937984 10104 tablet_bootstrap.cc:492] T 402e76e6f92744fbb36653d04f861e4f P 39981a8dfe94462b9a73ba489af4c714: Bootstrap complete.
I20240502 21:27:33.938300 10104 ts_tablet_manager.cc:1367] T 402e76e6f92744fbb36653d04f861e4f P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.046s user 0.009s sys 0.002s
I20240502 21:27:33.939107 10109 ts_tablet_manager.cc:1367] T 80304ab48274408393c2e245477c3a78 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.049s user 0.009s sys 0.005s
I20240502 21:27:33.940191 10109 raft_consensus.cc:348] T 80304ab48274408393c2e245477c3a78 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:33.940462 10109 raft_consensus.cc:729] T 80304ab48274408393c2e245477c3a78 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:33.940977 10104 raft_consensus.cc:348] T 402e76e6f92744fbb36653d04f861e4f P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:33.941246 10104 raft_consensus.cc:729] T 402e76e6f92744fbb36653d04f861e4f P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:33.943897 10104 consensus_queue.cc:260] T 402e76e6f92744fbb36653d04f861e4f P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:33.944375 10109 consensus_queue.cc:260] T 80304ab48274408393c2e245477c3a78 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:33.945021 10109 ts_tablet_manager.cc:1398] T 80304ab48274408393c2e245477c3a78 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 21:27:33.948443 10104 ts_tablet_manager.cc:1398] T 402e76e6f92744fbb36653d04f861e4f P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240502 21:27:33.950053 7505 tablet_copy_service.cc:342] P 31a72286397c4edc9a63eca3412be168: Request end of tablet copy session 39981a8dfe94462b9a73ba489af4c714-402e76e6f92744fbb36653d04f861e4f received from {username='slave'} at 127.29.82.69:38761
I20240502 21:27:33.950237 7505 tablet_copy_service.cc:433] P 31a72286397c4edc9a63eca3412be168: ending tablet copy session 39981a8dfe94462b9a73ba489af4c714-402e76e6f92744fbb36653d04f861e4f on tablet 402e76e6f92744fbb36653d04f861e4f with peer 39981a8dfe94462b9a73ba489af4c714
I20240502 21:27:33.954355 10133 ts_tablet_manager.cc:902] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c: Initiating tablet copy from peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:33.956895 7505 tablet_copy_service.cc:342] P 31a72286397c4edc9a63eca3412be168: Request end of tablet copy session 398c3c8f3195492c900439ad5e1931bb-80304ab48274408393c2e245477c3a78 received from {username='slave'} at 127.29.82.67:43117
I20240502 21:27:33.957054 7505 tablet_copy_service.cc:433] P 31a72286397c4edc9a63eca3412be168: ending tablet copy session 398c3c8f3195492c900439ad5e1931bb-80304ab48274408393c2e245477c3a78 on tablet 80304ab48274408393c2e245477c3a78 with peer 398c3c8f3195492c900439ad5e1931bb
I20240502 21:27:33.976553 10133 tablet_copy_client.cc:308] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.65:42781
I20240502 21:27:33.977442 4463 tablet_copy_service.cc:140] P 71e1375b799b439fa3c83ca763c039c1: Received BeginTabletCopySession request for tablet 1c243b27429043aa8642d9b2ed595ab3 from peer d0456c5e55a542128688904e003d403c ({username='slave'} at 127.29.82.70:41689)
I20240502 21:27:33.977615 4463 tablet_copy_service.cc:161] P 71e1375b799b439fa3c83ca763c039c1: Beginning new tablet copy session on tablet 1c243b27429043aa8642d9b2ed595ab3 from peer d0456c5e55a542128688904e003d403c at {username='slave'} at 127.29.82.70:41689: session id = d0456c5e55a542128688904e003d403c-1c243b27429043aa8642d9b2ed595ab3
I20240502 21:27:33.979377 4839 consensus_queue.cc:237] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:33.980182 4463 tablet_copy_source_session.cc:205] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:33.982204 7350 raft_consensus.cc:1261] T acdc893a70224a7b82083e998910d6ef P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:33.982255 7484 raft_consensus.cc:1261] T acdc893a70224a7b82083e998910d6ef P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:33.982733 9522 consensus_queue.cc:1035] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:27:33.983036 4773 consensus_peers.cc:485] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: acdc893a70224a7b82083e998910d6ef. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:33.983187 9522 consensus_queue.cc:1035] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:33.984645 10133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c243b27429043aa8642d9b2ed595ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:33.987504 10132 raft_consensus.cc:2932] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:33.993752 7484 raft_consensus.cc:2932] T acdc893a70224a7b82083e998910d6ef P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:33.998148 7348 raft_consensus.cc:2932] T acdc893a70224a7b82083e998910d6ef P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:34.001807 4299 catalog_manager.cc:5484] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:34.004896 10133 tablet_copy_client.cc:783] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:34.005297 10133 tablet_copy_client.cc:647] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:34.009260 10133 tablet_copy_client.cc:523] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:34.013701 10133 tablet_bootstrap.cc:492] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:34.064807 10096 ts_tablet_manager.cc:902] T d6370cfa3d804e0fa5dc2a0f71d78840 P 73a1f2ca1f544de9b0e3b37015f85328: Initiating tablet copy from peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:27:34.086409 10096 tablet_copy_client.cc:308] T d6370cfa3d804e0fa5dc2a0f71d78840 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.66:35781
I20240502 21:27:34.087342 4595 tablet_copy_service.cc:140] P 7a736a8ebb164ce0a051c0fccd76c3b8: Received BeginTabletCopySession request for tablet d6370cfa3d804e0fa5dc2a0f71d78840 from peer 73a1f2ca1f544de9b0e3b37015f85328 ({username='slave'} at 127.29.82.68:46725)
I20240502 21:27:34.087500 4595 tablet_copy_service.cc:161] P 7a736a8ebb164ce0a051c0fccd76c3b8: Beginning new tablet copy session on tablet d6370cfa3d804e0fa5dc2a0f71d78840 from peer 73a1f2ca1f544de9b0e3b37015f85328 at {username='slave'} at 127.29.82.68:46725: session id = 73a1f2ca1f544de9b0e3b37015f85328-d6370cfa3d804e0fa5dc2a0f71d78840
I20240502 21:27:34.088202 10133 tablet_bootstrap.cc:492] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:34.088973 10133 tablet_bootstrap.cc:492] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c: Bootstrap complete.
I20240502 21:27:34.089584 10133 ts_tablet_manager.cc:1367] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.076s user 0.049s sys 0.014s
I20240502 21:27:34.091014 4595 tablet_copy_source_session.cc:205] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:34.091070 10133 raft_consensus.cc:348] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:34.091502 10133 raft_consensus.cc:729] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:34.091931 10133 consensus_queue.cc:260] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:34.092466 10096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6370cfa3d804e0fa5dc2a0f71d78840. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:34.092984 10133 ts_tablet_manager.cc:1398] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:27:34.093881 4463 tablet_copy_service.cc:342] P 71e1375b799b439fa3c83ca763c039c1: Request end of tablet copy session d0456c5e55a542128688904e003d403c-1c243b27429043aa8642d9b2ed595ab3 received from {username='slave'} at 127.29.82.70:41689
I20240502 21:27:34.094000 4463 tablet_copy_service.cc:433] P 71e1375b799b439fa3c83ca763c039c1: ending tablet copy session d0456c5e55a542128688904e003d403c-1c243b27429043aa8642d9b2ed595ab3 on tablet 1c243b27429043aa8642d9b2ed595ab3 with peer d0456c5e55a542128688904e003d403c
I20240502 21:27:34.098456 10096 tablet_copy_client.cc:783] T d6370cfa3d804e0fa5dc2a0f71d78840 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:34.098685 10096 tablet_copy_client.cc:647] T d6370cfa3d804e0fa5dc2a0f71d78840 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:34.100638 10096 tablet_copy_client.cc:523] T d6370cfa3d804e0fa5dc2a0f71d78840 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:34.104596 10096 tablet_bootstrap.cc:492] T d6370cfa3d804e0fa5dc2a0f71d78840 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:34.106360 7616 raft_consensus.cc:1203] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:34.111152 10096 tablet_bootstrap.cc:492] T d6370cfa3d804e0fa5dc2a0f71d78840 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:34.111506 10096 tablet_bootstrap.cc:492] T d6370cfa3d804e0fa5dc2a0f71d78840 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap complete.
I20240502 21:27:34.111764 10096 ts_tablet_manager.cc:1367] T d6370cfa3d804e0fa5dc2a0f71d78840 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240502 21:27:34.112505 10096 raft_consensus.cc:348] T d6370cfa3d804e0fa5dc2a0f71d78840 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:34.112730 10096 raft_consensus.cc:729] T d6370cfa3d804e0fa5dc2a0f71d78840 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:34.112924 10096 consensus_queue.cc:260] T d6370cfa3d804e0fa5dc2a0f71d78840 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:34.113529 10096 ts_tablet_manager.cc:1398] T d6370cfa3d804e0fa5dc2a0f71d78840 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:34.118867 4837 raft_consensus.cc:1203] T 4cc82d5106804490993007f9f75943aa P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:34.121294 4595 tablet_copy_service.cc:342] P 7a736a8ebb164ce0a051c0fccd76c3b8: Request end of tablet copy session 73a1f2ca1f544de9b0e3b37015f85328-d6370cfa3d804e0fa5dc2a0f71d78840 received from {username='slave'} at 127.29.82.68:46725
I20240502 21:27:34.121429 4595 tablet_copy_service.cc:433] P 7a736a8ebb164ce0a051c0fccd76c3b8: ending tablet copy session 73a1f2ca1f544de9b0e3b37015f85328-d6370cfa3d804e0fa5dc2a0f71d78840 on tablet d6370cfa3d804e0fa5dc2a0f71d78840 with peer 73a1f2ca1f544de9b0e3b37015f85328
I20240502 21:27:34.135183 7482 consensus_queue.cc:237] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:34.138271 4839 raft_consensus.cc:1261] T adbccf5386624b28af36ff893ac9378f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:27:34.138463 7419 consensus_peers.cc:485] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 -> Peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Couldn't send request to peer d0456c5e55a542128688904e003d403c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: adbccf5386624b28af36ff893ac9378f. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:34.138657 4574 raft_consensus.cc:1261] T adbccf5386624b28af36ff893ac9378f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:34.138814 9672 consensus_queue.cc:1035] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:34.139999 9672 consensus_queue.cc:1035] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:34.143633 10006 raft_consensus.cc:2932] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:34.144368 4839 raft_consensus.cc:2932] T adbccf5386624b28af36ff893ac9378f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:34.150138 4575 raft_consensus.cc:2932] T adbccf5386624b28af36ff893ac9378f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:34.153563 4703 raft_consensus.cc:1203] T 80304ab48274408393c2e245477c3a78 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:34.153625 4299 catalog_manager.cc:5484] T adbccf5386624b28af36ff893ac9378f P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index -1 to 2, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:34.225993 10109 ts_tablet_manager.cc:902] T 81db5e77d00446cba3eb8345ee3df476 P 398c3c8f3195492c900439ad5e1931bb: Initiating tablet copy from peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:34.241372 4574 raft_consensus.cc:1203] T 37d864bed6ef46e4a3f57de0373bbe57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:34.259601 10191 ts_tablet_manager.cc:902] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04: Initiating tablet copy from peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:27:34.278280 10109 tablet_copy_client.cc:308] T 81db5e77d00446cba3eb8345ee3df476 P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.65:42781
I20240502 21:27:34.280416 4463 tablet_copy_service.cc:140] P 71e1375b799b439fa3c83ca763c039c1: Received BeginTabletCopySession request for tablet 81db5e77d00446cba3eb8345ee3df476 from peer 398c3c8f3195492c900439ad5e1931bb ({username='slave'} at 127.29.82.67:49529)
I20240502 21:27:34.280644 4463 tablet_copy_service.cc:161] P 71e1375b799b439fa3c83ca763c039c1: Beginning new tablet copy session on tablet 81db5e77d00446cba3eb8345ee3df476 from peer 398c3c8f3195492c900439ad5e1931bb at {username='slave'} at 127.29.82.67:49529: session id = 398c3c8f3195492c900439ad5e1931bb-81db5e77d00446cba3eb8345ee3df476
I20240502 21:27:34.300323 7212 raft_consensus.cc:1203] T 402e76e6f92744fbb36653d04f861e4f P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:34.338589 4463 tablet_copy_source_session.cc:205] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:34.340216 10109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81db5e77d00446cba3eb8345ee3df476. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:34.348426 10109 tablet_copy_client.cc:783] T 81db5e77d00446cba3eb8345ee3df476 P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:34.348708 10109 tablet_copy_client.cc:647] T 81db5e77d00446cba3eb8345ee3df476 P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:34.350638 10109 tablet_copy_client.cc:523] T 81db5e77d00446cba3eb8345ee3df476 P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:34.355105 10109 tablet_bootstrap.cc:492] T 81db5e77d00446cba3eb8345ee3df476 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:34.388806 10109 tablet_bootstrap.cc:492] T 81db5e77d00446cba3eb8345ee3df476 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:34.389225 10109 tablet_bootstrap.cc:492] T 81db5e77d00446cba3eb8345ee3df476 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap complete.
I20240502 21:27:34.389515 10109 ts_tablet_manager.cc:1367] T 81db5e77d00446cba3eb8345ee3df476 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.002s
I20240502 21:27:34.390424 10109 raft_consensus.cc:348] T 81db5e77d00446cba3eb8345ee3df476 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:34.390740 10109 raft_consensus.cc:729] T 81db5e77d00446cba3eb8345ee3df476 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:34.390974 10109 consensus_queue.cc:260] T 81db5e77d00446cba3eb8345ee3df476 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:34.391551 10109 ts_tablet_manager.cc:1398] T 81db5e77d00446cba3eb8345ee3df476 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:34.391896 10191 tablet_copy_client.cc:308] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.67:34537
I20240502 21:27:34.393215 4727 tablet_copy_service.cc:140] P 398c3c8f3195492c900439ad5e1931bb: Received BeginTabletCopySession request for tablet 2285212d78b0460ca329949eb3140e69 from peer 0ddd486109f5434cace13d60d633ac04 ({username='slave'} at 127.29.82.72:33287)
I20240502 21:27:34.393450 4727 tablet_copy_service.cc:161] P 398c3c8f3195492c900439ad5e1931bb: Beginning new tablet copy session on tablet 2285212d78b0460ca329949eb3140e69 from peer 0ddd486109f5434cace13d60d633ac04 at {username='slave'} at 127.29.82.72:33287: session id = 0ddd486109f5434cace13d60d633ac04-2285212d78b0460ca329949eb3140e69
I20240502 21:27:34.397974 4463 tablet_copy_service.cc:342] P 71e1375b799b439fa3c83ca763c039c1: Request end of tablet copy session 398c3c8f3195492c900439ad5e1931bb-81db5e77d00446cba3eb8345ee3df476 received from {username='slave'} at 127.29.82.67:49529
I20240502 21:27:34.398126 4463 tablet_copy_service.cc:433] P 71e1375b799b439fa3c83ca763c039c1: ending tablet copy session 398c3c8f3195492c900439ad5e1931bb-81db5e77d00446cba3eb8345ee3df476 on tablet 81db5e77d00446cba3eb8345ee3df476 with peer 398c3c8f3195492c900439ad5e1931bb
I20240502 21:27:34.420904 4727 tablet_copy_source_session.cc:205] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:34.422884 10191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2285212d78b0460ca329949eb3140e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:34.429725 10191 tablet_copy_client.cc:783] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:34.430019 10191 tablet_copy_client.cc:647] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:34.433116 10191 tablet_copy_client.cc:523] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:34.436298 10191 tablet_bootstrap.cc:492] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:34.466485 10208 ts_tablet_manager.cc:902] T acdc893a70224a7b82083e998910d6ef P 7a736a8ebb164ce0a051c0fccd76c3b8: Initiating tablet copy from peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:34.473183 7350 raft_consensus.cc:1203] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:34.484356 10208 tablet_copy_client.cc:308] T acdc893a70224a7b82083e998910d6ef P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.68:42449
I20240502 21:27:34.485051 4859 tablet_copy_service.cc:140] P 73a1f2ca1f544de9b0e3b37015f85328: Received BeginTabletCopySession request for tablet acdc893a70224a7b82083e998910d6ef from peer 7a736a8ebb164ce0a051c0fccd76c3b8 ({username='slave'} at 127.29.82.66:35827)
I20240502 21:27:34.485203 4859 tablet_copy_service.cc:161] P 73a1f2ca1f544de9b0e3b37015f85328: Beginning new tablet copy session on tablet acdc893a70224a7b82083e998910d6ef from peer 7a736a8ebb164ce0a051c0fccd76c3b8 at {username='slave'} at 127.29.82.66:35827: session id = 7a736a8ebb164ce0a051c0fccd76c3b8-acdc893a70224a7b82083e998910d6ef
I20240502 21:27:34.491052 4859 tablet_copy_source_session.cc:205] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:34.492705 10208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acdc893a70224a7b82083e998910d6ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:34.501547 10208 tablet_copy_client.cc:783] T acdc893a70224a7b82083e998910d6ef P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:34.501838 10208 tablet_copy_client.cc:647] T acdc893a70224a7b82083e998910d6ef P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:34.508809 10208 tablet_copy_client.cc:523] T acdc893a70224a7b82083e998910d6ef P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:34.513494 10208 tablet_bootstrap.cc:492] T acdc893a70224a7b82083e998910d6ef P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:34.542083 10191 tablet_bootstrap.cc:492] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:34.542708 10191 tablet_bootstrap.cc:492] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04: Bootstrap complete.
I20240502 21:27:34.543107 10191 ts_tablet_manager.cc:1367] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.107s user 0.054s sys 0.002s
I20240502 21:27:34.544163 10191 raft_consensus.cc:348] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:34.544320 4839 raft_consensus.cc:1203] T d6370cfa3d804e0fa5dc2a0f71d78840 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:34.544569 10191 raft_consensus.cc:729] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:34.544934 10191 consensus_queue.cc:260] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:34.545974 10191 ts_tablet_manager.cc:1398] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:34.546816 4727 tablet_copy_service.cc:342] P 398c3c8f3195492c900439ad5e1931bb: Request end of tablet copy session 0ddd486109f5434cace13d60d633ac04-2285212d78b0460ca329949eb3140e69 received from {username='slave'} at 127.29.82.72:33287
I20240502 21:27:34.547011 4727 tablet_copy_service.cc:433] P 398c3c8f3195492c900439ad5e1931bb: ending tablet copy session 0ddd486109f5434cace13d60d633ac04-2285212d78b0460ca329949eb3140e69 on tablet 2285212d78b0460ca329949eb3140e69 with peer 0ddd486109f5434cace13d60d633ac04
I20240502 21:27:34.552803 10208 tablet_bootstrap.cc:492] T acdc893a70224a7b82083e998910d6ef P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:34.553303 10208 tablet_bootstrap.cc:492] T acdc893a70224a7b82083e998910d6ef P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap complete.
I20240502 21:27:34.553628 10208 ts_tablet_manager.cc:1367] T acdc893a70224a7b82083e998910d6ef P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.006s
I20240502 21:27:34.554486 10208 raft_consensus.cc:348] T acdc893a70224a7b82083e998910d6ef P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:34.555137 10208 raft_consensus.cc:729] T acdc893a70224a7b82083e998910d6ef P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:34.555435 10208 consensus_queue.cc:260] T acdc893a70224a7b82083e998910d6ef P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:34.556226 10208 ts_tablet_manager.cc:1398] T acdc893a70224a7b82083e998910d6ef P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:34.566828 4859 tablet_copy_service.cc:342] P 73a1f2ca1f544de9b0e3b37015f85328: Request end of tablet copy session 7a736a8ebb164ce0a051c0fccd76c3b8-acdc893a70224a7b82083e998910d6ef received from {username='slave'} at 127.29.82.66:35827
I20240502 21:27:34.566968 4859 tablet_copy_service.cc:433] P 73a1f2ca1f544de9b0e3b37015f85328: ending tablet copy session 7a736a8ebb164ce0a051c0fccd76c3b8-acdc893a70224a7b82083e998910d6ef on tablet acdc893a70224a7b82083e998910d6ef with peer 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:27:34.615219 10229 ts_tablet_manager.cc:902] T adbccf5386624b28af36ff893ac9378f P d0456c5e55a542128688904e003d403c: Initiating tablet copy from peer 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:34.654309 10229 tablet_copy_client.cc:308] T adbccf5386624b28af36ff893ac9378f P d0456c5e55a542128688904e003d403c: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.71:33913
I20240502 21:27:34.654984 7505 tablet_copy_service.cc:140] P 31a72286397c4edc9a63eca3412be168: Received BeginTabletCopySession request for tablet adbccf5386624b28af36ff893ac9378f from peer d0456c5e55a542128688904e003d403c ({username='slave'} at 127.29.82.70:39533)
I20240502 21:27:34.655170 7505 tablet_copy_service.cc:161] P 31a72286397c4edc9a63eca3412be168: Beginning new tablet copy session on tablet adbccf5386624b28af36ff893ac9378f from peer d0456c5e55a542128688904e003d403c at {username='slave'} at 127.29.82.70:39533: session id = d0456c5e55a542128688904e003d403c-adbccf5386624b28af36ff893ac9378f
I20240502 21:27:34.658218 7505 tablet_copy_source_session.cc:205] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:34.660123 10229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adbccf5386624b28af36ff893ac9378f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:34.665150 4703 raft_consensus.cc:1203] T 81db5e77d00446cba3eb8345ee3df476 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:34.679237 10229 tablet_copy_client.cc:783] T adbccf5386624b28af36ff893ac9378f P d0456c5e55a542128688904e003d403c: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:34.679551 10229 tablet_copy_client.cc:647] T adbccf5386624b28af36ff893ac9378f P d0456c5e55a542128688904e003d403c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:34.685772 10229 tablet_copy_client.cc:523] T adbccf5386624b28af36ff893ac9378f P d0456c5e55a542128688904e003d403c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:34.690052 10229 tablet_bootstrap.cc:492] T adbccf5386624b28af36ff893ac9378f P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:34.697281 10229 tablet_bootstrap.cc:492] T adbccf5386624b28af36ff893ac9378f P d0456c5e55a542128688904e003d403c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:34.697933 10229 tablet_bootstrap.cc:492] T adbccf5386624b28af36ff893ac9378f P d0456c5e55a542128688904e003d403c: Bootstrap complete.
I20240502 21:27:34.698268 10229 ts_tablet_manager.cc:1367] T adbccf5386624b28af36ff893ac9378f P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20240502 21:27:34.699155 10229 raft_consensus.cc:348] T adbccf5386624b28af36ff893ac9378f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:34.699488 10229 raft_consensus.cc:729] T adbccf5386624b28af36ff893ac9378f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:34.699868 10229 consensus_queue.cc:260] T adbccf5386624b28af36ff893ac9378f P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:34.700677 10229 ts_tablet_manager.cc:1398] T adbccf5386624b28af36ff893ac9378f P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:34.701385 7505 tablet_copy_service.cc:342] P 31a72286397c4edc9a63eca3412be168: Request end of tablet copy session d0456c5e55a542128688904e003d403c-adbccf5386624b28af36ff893ac9378f received from {username='slave'} at 127.29.82.70:39533
I20240502 21:27:34.701524 7505 tablet_copy_service.cc:433] P 31a72286397c4edc9a63eca3412be168: ending tablet copy session d0456c5e55a542128688904e003d403c-adbccf5386624b28af36ff893ac9378f on tablet adbccf5386624b28af36ff893ac9378f with peer d0456c5e55a542128688904e003d403c
I20240502 21:27:34.830853 7618 raft_consensus.cc:1203] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:34.850901 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:34.878978 4574 raft_consensus.cc:1203] T acdc893a70224a7b82083e998910d6ef P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:34.879964 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:34.968597 4555 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 21:27:34.978354 7151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.67:34537, user_credentials={real_user=slave}} blocked reactor thread for 44632.5us
I20240502 21:27:34.984127 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 21:27:35.020529 7150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.70:40719, user_credentials={real_user=slave}} blocked reactor thread for 49040.3us
W20240502 21:27:35.057006 7149 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.72:41223, user_credentials={real_user=slave}} blocked reactor thread for 123539us
I20240502 21:27:35.059120 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:35.060062 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:27:35.096496 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 21:27:35.102146 4642 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.66:35781, user_credentials={real_user=slave}} blocked reactor thread for 57670us
I20240502 21:27:35.114492 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:35.136216 7348 raft_consensus.cc:1203] T adbccf5386624b28af36ff893ac9378f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:35.252761 4837 consensus_queue.cc:237] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:35.256246 4575 raft_consensus.cc:1261] T acdc893a70224a7b82083e998910d6ef P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:35.256481 7350 raft_consensus.cc:1261] T acdc893a70224a7b82083e998910d6ef P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:35.257041 10132 consensus_queue.cc:1035] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:35.257265 9522 consensus_queue.cc:1035] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:35.260856 9522 raft_consensus.cc:2932] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:35.266904 4575 raft_consensus.cc:2932] T acdc893a70224a7b82083e998910d6ef P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:35.267328 7350 raft_consensus.cc:2932] T acdc893a70224a7b82083e998910d6ef P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:35.272130 4299 catalog_manager.cc:5484] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:35.295596 7462 tablet_service.cc:1514] Processing DeleteTablet for tablet acdc893a70224a7b82083e998910d6ef with delete_type TABLET_DATA_TOMBSTONED (TS 31a72286397c4edc9a63eca3412be168 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:37182
I20240502 21:27:35.300145 10300 tablet_replica.cc:330] T acdc893a70224a7b82083e998910d6ef P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:27:35.300735 10300 raft_consensus.cc:2229] T acdc893a70224a7b82083e998910d6ef P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:35.301002 10300 raft_consensus.cc:2258] T acdc893a70224a7b82083e998910d6ef P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:35.301733 10300 ts_tablet_manager.cc:1875] T acdc893a70224a7b82083e998910d6ef P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:35.320111 10300 ts_tablet_manager.cc:1888] T acdc893a70224a7b82083e998910d6ef P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:27:35.323560 10300 log.cc:1196] T acdc893a70224a7b82083e998910d6ef P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/acdc893a70224a7b82083e998910d6ef
I20240502 21:27:35.324882 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet acdc893a70224a7b82083e998910d6ef (table rebalancer_extra_table_16 [id=23c1b78fb75948c193bd4eb1c32f8bef]) successfully deleted
I20240502 21:27:35.615588 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:35.629962 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:35.647037 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:35.693503 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:35.694082 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:35.730593 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:35.730630 4555 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 21:27:35.805666 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 21:27:35.911365 7416 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.65:42781, user_credentials={real_user=slave}} blocked reactor thread for 45425.7us
I20240502 21:27:35.990487 4705 consensus_queue.cc:237] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:35.992465 4574 raft_consensus.cc:1261] T 2285212d78b0460ca329949eb3140e69 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:35.992628 7616 raft_consensus.cc:1261] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:35.993175 10271 consensus_queue.cc:1035] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:35.993692 10271 consensus_queue.cc:1035] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:35.996732 10075 raft_consensus.cc:2932] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:35.999991 7616 raft_consensus.cc:2932] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:36.000500 4575 raft_consensus.cc:2932] T 2285212d78b0460ca329949eb3140e69 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:36.006217 4293 catalog_manager.cc:5484] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: config changed from index 4 to 5, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) evicted. New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:36.018150 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet 2285212d78b0460ca329949eb3140e69 with delete_type TABLET_DATA_TOMBSTONED (TS 71e1375b799b439fa3c83ca763c039c1 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:36.018936 10384 tablet_replica.cc:330] T 2285212d78b0460ca329949eb3140e69 P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:36.019212 10384 raft_consensus.cc:2229] T 2285212d78b0460ca329949eb3140e69 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:36.021239 10384 raft_consensus.cc:2258] T 2285212d78b0460ca329949eb3140e69 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:36.023840 10384 ts_tablet_manager.cc:1875] T 2285212d78b0460ca329949eb3140e69 P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:36.042932 10384 ts_tablet_manager.cc:1888] T 2285212d78b0460ca329949eb3140e69 P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:27:36.043179 10384 log.cc:1196] T 2285212d78b0460ca329949eb3140e69 P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/2285212d78b0460ca329949eb3140e69
I20240502 21:27:36.048770 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet 2285212d78b0460ca329949eb3140e69 (table rebalance_test_table_7 [id=ccfdd338c3a14e4eaab745d78ee6de47]) successfully deleted
I20240502 21:27:36.390349 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:36.443495 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:36.469249 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:36.472185 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:36.537526 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 21:27:36.545039 4508 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.68:42449, user_credentials={real_user=slave}} blocked reactor thread for 95377.9us
I20240502 21:27:36.562474 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:36.610013 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:36.631508 4555 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 21:27:36.740619 7616 consensus_queue.cc:237] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:36.742486 4575 raft_consensus.cc:1261] T e8f42cdee5c24db982392d2057b588a2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:36.742560 4440 raft_consensus.cc:1261] T e8f42cdee5c24db982392d2057b588a2 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:36.743125 10239 consensus_queue.cc:1035] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:36.743438 9457 consensus_queue.cc:1035] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:36.746295 10239 raft_consensus.cc:2932] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:36.746938 4575 raft_consensus.cc:2932] T e8f42cdee5c24db982392d2057b588a2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:36.755506 4298 catalog_manager.cc:5484] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:36.759382 4440 raft_consensus.cc:2932] T e8f42cdee5c24db982392d2057b588a2 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:36.789786 7194 tablet_service.cc:1514] Processing DeleteTablet for tablet e8f42cdee5c24db982392d2057b588a2 with delete_type TABLET_DATA_TOMBSTONED (TS 39981a8dfe94462b9a73ba489af4c714 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44652
I20240502 21:27:36.801851 10478 tablet_replica.cc:330] T e8f42cdee5c24db982392d2057b588a2 P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:27:36.802306 10478 raft_consensus.cc:2229] T e8f42cdee5c24db982392d2057b588a2 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:36.802722 10478 raft_consensus.cc:2258] T e8f42cdee5c24db982392d2057b588a2 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:36.803548 10478 ts_tablet_manager.cc:1875] T e8f42cdee5c24db982392d2057b588a2 P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:36.814375 10478 ts_tablet_manager.cc:1888] T e8f42cdee5c24db982392d2057b588a2 P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:27:36.814610 10478 log.cc:1196] T e8f42cdee5c24db982392d2057b588a2 P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/e8f42cdee5c24db982392d2057b588a2
I20240502 21:27:36.815608 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet e8f42cdee5c24db982392d2057b588a2 (table rebalancer_extra_table_0 [id=ab7a7a9b6e69455484db6423f42c7c06]) successfully deleted
I20240502 21:27:37.259814 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:37.262563 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:37.266151 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:37.310642 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:37.311708 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:37.347347 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:27:37.380385 4555 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 21:27:37.401906 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:37.574302 7212 consensus_queue.cc:237] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:37.576571 4575 raft_consensus.cc:1261] T 030fa30553294e51808df57d6bb70b66 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:37.576665 4442 raft_consensus.cc:1261] T 030fa30553294e51808df57d6bb70b66 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:37.577132 10451 consensus_queue.cc:1035] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:37.577459 10265 consensus_queue.cc:1035] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:37.580405 10265 raft_consensus.cc:2932] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:37.598800 4294 catalog_manager.cc:5484] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:37.609028 4574 raft_consensus.cc:2932] T 030fa30553294e51808df57d6bb70b66 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:37.611104 4442 raft_consensus.cc:2932] T 030fa30553294e51808df57d6bb70b66 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:37.625504 7328 tablet_service.cc:1514] Processing DeleteTablet for tablet 030fa30553294e51808df57d6bb70b66 with delete_type TABLET_DATA_TOMBSTONED (TS d0456c5e55a542128688904e003d403c not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47078
I20240502 21:27:37.627516 10578 tablet_replica.cc:330] T 030fa30553294e51808df57d6bb70b66 P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:27:37.627913 10578 raft_consensus.cc:2229] T 030fa30553294e51808df57d6bb70b66 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:37.630400 10578 raft_consensus.cc:2258] T 030fa30553294e51808df57d6bb70b66 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:37.631184 10578 ts_tablet_manager.cc:1875] T 030fa30553294e51808df57d6bb70b66 P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:37.639225 10578 ts_tablet_manager.cc:1888] T 030fa30553294e51808df57d6bb70b66 P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:27:37.639405 10578 log.cc:1196] T 030fa30553294e51808df57d6bb70b66 P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/030fa30553294e51808df57d6bb70b66
I20240502 21:27:37.640583 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet 030fa30553294e51808df57d6bb70b66 (table rebalancer_extra_table_20 [id=8a6e0f9c1e92442cbf43f55d3eef080f]) successfully deleted
I20240502 21:27:38.013962 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:38.032404 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:38.087262 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:38.102023 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:38.128810 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:38.133929 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:38.168923 4555 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 21:27:38.199996 4772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.69:33193, user_credentials={real_user=slave}} blocked reactor thread for 139128us
I20240502 21:27:38.267547 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:27:38.442088 4839 consensus_queue.cc:237] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:38.444366 4574 raft_consensus.cc:1261] T 86fa4255c6554ba28e141de3cb328976 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:38.444414 7618 raft_consensus.cc:1261] T 86fa4255c6554ba28e141de3cb328976 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:38.444916 10646 consensus_queue.cc:1035] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:38.445192 10643 consensus_queue.cc:1035] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:38.448035 10643 raft_consensus.cc:2932] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:38.448689 7618 raft_consensus.cc:2932] T 86fa4255c6554ba28e141de3cb328976 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:38.449314 4574 raft_consensus.cc:2932] T 86fa4255c6554ba28e141de3cb328976 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:38.454684 4296 catalog_manager.cc:5484] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:38.462131 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet 86fa4255c6554ba28e141de3cb328976 with delete_type TABLET_DATA_TOMBSTONED (TS 398c3c8f3195492c900439ad5e1931bb not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44764
I20240502 21:27:38.463470 10677 tablet_replica.cc:330] T 86fa4255c6554ba28e141de3cb328976 P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:27:38.463758 10677 raft_consensus.cc:2229] T 86fa4255c6554ba28e141de3cb328976 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:38.464136 10677 raft_consensus.cc:2258] T 86fa4255c6554ba28e141de3cb328976 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:38.465041 10677 ts_tablet_manager.cc:1875] T 86fa4255c6554ba28e141de3cb328976 P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:38.470824 10677 ts_tablet_manager.cc:1888] T 86fa4255c6554ba28e141de3cb328976 P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:27:38.470999 10677 log.cc:1196] T 86fa4255c6554ba28e141de3cb328976 P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/86fa4255c6554ba28e141de3cb328976
I20240502 21:27:38.471712 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet 86fa4255c6554ba28e141de3cb328976 (table rebalance_test_table_1 [id=d6c511862aad4ba5a222ddedb7e95c27]) successfully deleted
I20240502 21:27:38.756477 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:38.773201 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:38.806736 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:38.838219 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:38.882088 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:38.912552 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:27:38.913204 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:38.967931 4555 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 21:27:39.062937 7350 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fed543e42f844227a14a8d812ce158ba"
dest_uuid: "d0456c5e55a542128688904e003d403c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51282
I20240502 21:27:39.063189 7350 raft_consensus.cc:595] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Received request to transfer leadership
I20240502 21:27:39.393527 10450 raft_consensus.cc:979] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c: : Instructing follower 7a736a8ebb164ce0a051c0fccd76c3b8 to start an election
I20240502 21:27:39.394510 10792 raft_consensus.cc:1067] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Signalling peer 7a736a8ebb164ce0a051c0fccd76c3b8 to start an election
I20240502 21:27:39.395949 4575 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fed543e42f844227a14a8d812ce158ba"
dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
from {username='slave'} at 127.29.82.70:46121
I20240502 21:27:39.396265 4575 raft_consensus.cc:482] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.396379 4575 raft_consensus.cc:3037] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:39.399359 4575 raft_consensus.cc:504] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:39.401037 7482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fed543e42f844227a14a8d812ce158ba" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:27:39.401147 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fed543e42f844227a14a8d812ce158ba" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:27:39.401307 7482 raft_consensus.cc:3037] T fed543e42f844227a14a8d812ce158ba P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:39.401393 7350 raft_consensus.cc:3032] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:39.401494 7350 raft_consensus.cc:729] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Running, Role: LEADER
I20240502 21:27:39.401795 7350 consensus_queue.cc:260] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:39.402289 7350 raft_consensus.cc:3037] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:39.405936 7482 raft_consensus.cc:2454] T fed543e42f844227a14a8d812ce158ba P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 2.
I20240502 21:27:39.406101 7350 raft_consensus.cc:2454] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 2.
I20240502 21:27:39.406729 4508 leader_election.cc:302] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 7a736a8ebb164ce0a051c0fccd76c3b8, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:27:39.407130 4575 leader_election.cc:288] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 election: Requested vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:27:39.408869 4704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fed543e42f844227a14a8d812ce158ba" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:27:39.409094 4704 raft_consensus.cc:3037] T fed543e42f844227a14a8d812ce158ba P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:39.411846 10441 raft_consensus.cc:2781] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:39.411983 10441 raft_consensus.cc:686] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 2 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:27:39.412247 10441 consensus_queue.cc:237] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:39.414438 4704 raft_consensus.cc:2454] T fed543e42f844227a14a8d812ce158ba P 398c3c8f3195492c900439ad5e1931bb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 2.
I20240502 21:27:39.426755 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:39.444973 4299 catalog_manager.cc:5484] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 1 to 2, leader changed from d0456c5e55a542128688904e003d403c (127.29.82.70) to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 2 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:39.473860 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:39.474814 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:27:39.492233 7328 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 21:27:39.548470 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:39.557245 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:39.567787 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:39.605728 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:39.807946 7482 raft_consensus.cc:1261] T fed543e42f844227a14a8d812ce158ba P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 21:27:39.808604 10790 consensus_queue.cc:1035] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:39.954036 7350 raft_consensus.cc:1261] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 21:27:39.954625 10441 consensus_queue.cc:1035] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:39.984220 4704 raft_consensus.cc:1261] T fed543e42f844227a14a8d812ce158ba P 398c3c8f3195492c900439ad5e1931bb [term 2 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 21:27:39.989529 10441 consensus_queue.cc:1035] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:40.121187 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:40.125914 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:40.143371 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:40.152719 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:40.164999 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:27:40.168931 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:40.193156 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:40.232933 7328 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240502 21:27:40.235911 4642 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.66:35781, user_credentials={real_user=slave}} blocked reactor thread for 59769.1us
I20240502 21:27:40.434352 4837 consensus_queue.cc:237] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:40.436719 4443 raft_consensus.cc:1261] T 202f8b10d5674afa875da2f3929bd8da P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:40.436856 7616 raft_consensus.cc:1261] T 202f8b10d5674afa875da2f3929bd8da P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:40.437259 9522 consensus_queue.cc:1035] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:40.437536 10815 consensus_queue.cc:1035] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:40.440261 10815 raft_consensus.cc:2932] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:40.440834 4443 raft_consensus.cc:2932] T 202f8b10d5674afa875da2f3929bd8da P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:40.448715 4296 catalog_manager.cc:5484] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:40.453115 7616 raft_consensus.cc:2932] T 202f8b10d5674afa875da2f3929bd8da P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:40.467629 7194 tablet_service.cc:1514] Processing DeleteTablet for tablet 202f8b10d5674afa875da2f3929bd8da with delete_type TABLET_DATA_TOMBSTONED (TS 39981a8dfe94462b9a73ba489af4c714 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44652
I20240502 21:27:40.472718 10918 tablet_replica.cc:330] T 202f8b10d5674afa875da2f3929bd8da P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:27:40.473053 10918 raft_consensus.cc:2229] T 202f8b10d5674afa875da2f3929bd8da P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:40.473464 10918 raft_consensus.cc:2258] T 202f8b10d5674afa875da2f3929bd8da P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:40.474399 10918 ts_tablet_manager.cc:1875] T 202f8b10d5674afa875da2f3929bd8da P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:40.480876 10918 ts_tablet_manager.cc:1888] T 202f8b10d5674afa875da2f3929bd8da P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:27:40.483717 10918 log.cc:1196] T 202f8b10d5674afa875da2f3929bd8da P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/202f8b10d5674afa875da2f3929bd8da
I20240502 21:27:40.484802 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet 202f8b10d5674afa875da2f3929bd8da (table rebalancer_extra_table_60 [id=c758184ce285455e9f74bd2b6fd9dfc7]) successfully deleted
I20240502 21:27:40.739989 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:40.758522 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:40.775279 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:40.835470 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:40.856164 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:40.880651 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:27:40.965090 7328 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 21:27:41.000484 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:41.125916 4575 consensus_queue.cc:237] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:41.130438 4839 raft_consensus.cc:1261] T 796c7f3cc1f9426daecc89add3c6aa57 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:41.130690 7618 raft_consensus.cc:1261] T 796c7f3cc1f9426daecc89add3c6aa57 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:41.131053 10356 consensus_queue.cc:1035] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:41.131330 10441 consensus_queue.cc:1035] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:41.134443 10441 raft_consensus.cc:2932] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:41.135988 7618 raft_consensus.cc:2932] T 796c7f3cc1f9426daecc89add3c6aa57 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:41.136456 4837 raft_consensus.cc:2932] T 796c7f3cc1f9426daecc89add3c6aa57 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:41.143285 4299 catalog_manager.cc:5484] T 796c7f3cc1f9426daecc89add3c6aa57 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:41.160825 7462 tablet_service.cc:1514] Processing DeleteTablet for tablet 796c7f3cc1f9426daecc89add3c6aa57 with delete_type TABLET_DATA_TOMBSTONED (TS 31a72286397c4edc9a63eca3412be168 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:37182
I20240502 21:27:41.168227 11008 tablet_replica.cc:330] T 796c7f3cc1f9426daecc89add3c6aa57 P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:27:41.168576 11008 raft_consensus.cc:2229] T 796c7f3cc1f9426daecc89add3c6aa57 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:41.168960 11008 raft_consensus.cc:2258] T 796c7f3cc1f9426daecc89add3c6aa57 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:41.169678 11008 ts_tablet_manager.cc:1875] T 796c7f3cc1f9426daecc89add3c6aa57 P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:41.176303 11008 ts_tablet_manager.cc:1888] T 796c7f3cc1f9426daecc89add3c6aa57 P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:27:41.176559 11008 log.cc:1196] T 796c7f3cc1f9426daecc89add3c6aa57 P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/796c7f3cc1f9426daecc89add3c6aa57
I20240502 21:27:41.177372 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet 796c7f3cc1f9426daecc89add3c6aa57 (table rebalancer_extra_table_26 [id=1e6edd8bfe59407fa7863cd780bfae31]) successfully deleted
I20240502 21:27:41.589318 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:41.647621 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:41.708088 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:41.729739 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:41.740196 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:41.743618 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:27:41.751372 7328 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 21:27:41.837428 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:41.986976 4575 consensus_queue.cc:237] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:41.989645 7616 raft_consensus.cc:1261] T 4c06f3bc97254bff8181d3f6f02470b4 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:41.989771 4442 raft_consensus.cc:1261] T 4c06f3bc97254bff8181d3f6f02470b4 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:41.990451 10356 consensus_queue.cc:1035] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:41.990926 10356 consensus_queue.cc:1035] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:41.993274 10441 raft_consensus.cc:2932] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:41.994159 4442 raft_consensus.cc:2932] T 4c06f3bc97254bff8181d3f6f02470b4 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:41.994148 7616 raft_consensus.cc:2932] T 4c06f3bc97254bff8181d3f6f02470b4 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:42.000514 4295 catalog_manager.cc:5484] T 4c06f3bc97254bff8181d3f6f02470b4 P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:42.009191 7194 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c06f3bc97254bff8181d3f6f02470b4 with delete_type TABLET_DATA_TOMBSTONED (TS 39981a8dfe94462b9a73ba489af4c714 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44652
I20240502 21:27:42.010151 11108 tablet_replica.cc:330] T 4c06f3bc97254bff8181d3f6f02470b4 P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:27:42.010921 11108 raft_consensus.cc:2229] T 4c06f3bc97254bff8181d3f6f02470b4 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:42.011322 11108 raft_consensus.cc:2258] T 4c06f3bc97254bff8181d3f6f02470b4 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:42.012143 11108 ts_tablet_manager.cc:1875] T 4c06f3bc97254bff8181d3f6f02470b4 P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:42.017308 11108 ts_tablet_manager.cc:1888] T 4c06f3bc97254bff8181d3f6f02470b4 P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:27:42.017438 11108 log.cc:1196] T 4c06f3bc97254bff8181d3f6f02470b4 P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/4c06f3bc97254bff8181d3f6f02470b4
I20240502 21:27:42.018234 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet 4c06f3bc97254bff8181d3f6f02470b4 (table rebalancer_extra_table_38 [id=569b30a95e114b50ada524f1380f198e]) successfully deleted
I20240502 21:27:42.465104 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:42.504384 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:42.518545 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:42.538044 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:42.566700 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:42.602099 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:27:42.630044 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:42.651360 7328 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 21:27:42.851498 7618 consensus_queue.cc:237] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:42.853649 7350 raft_consensus.cc:1261] T 616eda3cb218475c8bf0108cc6b90810 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:42.853646 4837 raft_consensus.cc:1261] T 616eda3cb218475c8bf0108cc6b90810 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:42.854243 10239 consensus_queue.cc:1035] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:42.854996 10974 consensus_queue.cc:1035] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:42.857281 10239 raft_consensus.cc:2932] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:42.861133 4837 raft_consensus.cc:2932] T 616eda3cb218475c8bf0108cc6b90810 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:42.864389 4296 catalog_manager.cc:5484] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:42.873940 7350 raft_consensus.cc:2932] T 616eda3cb218475c8bf0108cc6b90810 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:42.911361 7462 tablet_service.cc:1514] Processing DeleteTablet for tablet 616eda3cb218475c8bf0108cc6b90810 with delete_type TABLET_DATA_TOMBSTONED (TS 31a72286397c4edc9a63eca3412be168 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:37182
I20240502 21:27:42.927206 11204 tablet_replica.cc:330] T 616eda3cb218475c8bf0108cc6b90810 P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:27:42.927449 11204 raft_consensus.cc:2229] T 616eda3cb218475c8bf0108cc6b90810 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:42.927812 11204 raft_consensus.cc:2258] T 616eda3cb218475c8bf0108cc6b90810 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:42.928622 11204 ts_tablet_manager.cc:1875] T 616eda3cb218475c8bf0108cc6b90810 P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:42.933959 11204 ts_tablet_manager.cc:1888] T 616eda3cb218475c8bf0108cc6b90810 P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:27:42.934185 11204 log.cc:1196] T 616eda3cb218475c8bf0108cc6b90810 P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/616eda3cb218475c8bf0108cc6b90810
I20240502 21:27:42.935351 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet 616eda3cb218475c8bf0108cc6b90810 (table rebalancer_extra_table_24 [id=add5d1abdc46419a86ab49c59629bc2d]) successfully deleted
I20240502 21:27:43.232699 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:43.259542 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:43.261145 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:43.264850 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:43.290633 7328 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 21:27:43.301187 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:43.353857 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:27:43.376156 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 21:27:43.459473 7150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.70:40719, user_credentials={real_user=slave}} blocked reactor thread for 61574.9us
I20240502 21:27:43.591840 4572 consensus_queue.cc:237] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:43.594141 4837 raft_consensus.cc:1261] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:43.594236 7618 raft_consensus.cc:1261] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:43.595160 10441 consensus_queue.cc:1035] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:43.595446 11170 consensus_queue.cc:1035] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:43.597966 10441 raft_consensus.cc:2932] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:43.599179 7617 raft_consensus.cc:2932] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:43.599193 4837 raft_consensus.cc:2932] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:43.605901 4299 catalog_manager.cc:5484] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:43.614426 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet c9fe2e18aa6c407fbbcbb74bf45bd520 with delete_type TABLET_DATA_TOMBSTONED (TS 398c3c8f3195492c900439ad5e1931bb not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44764
I20240502 21:27:43.615279 11294 tablet_replica.cc:330] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:27:43.615670 11294 raft_consensus.cc:2229] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:43.616006 11294 raft_consensus.cc:2258] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:43.617043 11294 ts_tablet_manager.cc:1875] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:43.625981 11294 ts_tablet_manager.cc:1888] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:27:43.626184 11294 log.cc:1196] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/c9fe2e18aa6c407fbbcbb74bf45bd520
I20240502 21:27:43.627970 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet c9fe2e18aa6c407fbbcbb74bf45bd520 (table rebalance_test_table_9 [id=80b493b0fc3f49dcbf948cee9e748e3c]) successfully deleted
I20240502 21:27:44.027364 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:44.070785 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:44.104311 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:44.111335 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:44.146368 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:27:44.193471 7328 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 21:27:44.200558 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:44.223105 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:44.410554 4440 consensus_queue.cc:237] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:44.413424 4839 raft_consensus.cc:1261] T a7a0167853fa420586455b82fb893c46 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:44.413769 7617 raft_consensus.cc:1261] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:44.414480 11262 consensus_queue.cc:1035] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:44.414768 11153 consensus_queue.cc:1035] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:44.425422 11153 raft_consensus.cc:2932] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:44.426317 7618 raft_consensus.cc:2932] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:44.433171 4293 catalog_manager.cc:5484] T a7a0167853fa420586455b82fb893c46 P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:44.435869 4839 raft_consensus.cc:2932] T a7a0167853fa420586455b82fb893c46 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:44.456027 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet a7a0167853fa420586455b82fb893c46 with delete_type TABLET_DATA_TOMBSTONED (TS 398c3c8f3195492c900439ad5e1931bb not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44764
I20240502 21:27:44.456682 11387 tablet_replica.cc:330] T a7a0167853fa420586455b82fb893c46 P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:27:44.456983 11387 raft_consensus.cc:2229] T a7a0167853fa420586455b82fb893c46 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:44.457355 11387 raft_consensus.cc:2258] T a7a0167853fa420586455b82fb893c46 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:44.458175 11387 ts_tablet_manager.cc:1875] T a7a0167853fa420586455b82fb893c46 P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:44.466284 11387 ts_tablet_manager.cc:1888] T a7a0167853fa420586455b82fb893c46 P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:27:44.466389 11387 log.cc:1196] T a7a0167853fa420586455b82fb893c46 P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/a7a0167853fa420586455b82fb893c46
I20240502 21:27:44.467123 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet a7a0167853fa420586455b82fb893c46 (table rebalance_test_table_15 [id=943341d7da6a4071b6595f437f13e626]) successfully deleted
I20240502 21:27:44.749045 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:44.769888 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:44.798259 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:44.816396 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:44.866055 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:44.866740 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:44.951545 7328 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 21:27:44.972859 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:27:45.090011 7483 consensus_queue.cc:237] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:45.093309 7349 raft_consensus.cc:1261] T adbccf5386624b28af36ff893ac9378f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:45.094014 11223 consensus_queue.cc:1035] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:45.095099 4572 raft_consensus.cc:1261] T adbccf5386624b28af36ff893ac9378f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:45.096127 11223 consensus_queue.cc:1035] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:45.100064 10980 raft_consensus.cc:2932] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:45.102874 4572 raft_consensus.cc:2932] T adbccf5386624b28af36ff893ac9378f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:45.107921 4293 catalog_manager.cc:5484] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 reported cstate change: config changed from index 2 to 3, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) evicted. New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:45.108500 7349 raft_consensus.cc:2932] T adbccf5386624b28af36ff893ac9378f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:45.121958 4817 tablet_service.cc:1514] Processing DeleteTablet for tablet adbccf5386624b28af36ff893ac9378f with delete_type TABLET_DATA_TOMBSTONED (TS 73a1f2ca1f544de9b0e3b37015f85328 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:56326
I20240502 21:27:45.122682 11475 tablet_replica.cc:330] T adbccf5386624b28af36ff893ac9378f P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:27:45.122959 11475 raft_consensus.cc:2229] T adbccf5386624b28af36ff893ac9378f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:45.123237 11475 raft_consensus.cc:2258] T adbccf5386624b28af36ff893ac9378f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:45.123836 11475 ts_tablet_manager.cc:1875] T adbccf5386624b28af36ff893ac9378f P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:45.130803 11475 ts_tablet_manager.cc:1888] T adbccf5386624b28af36ff893ac9378f P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:27:45.130935 11475 log.cc:1196] T adbccf5386624b28af36ff893ac9378f P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/adbccf5386624b28af36ff893ac9378f
I20240502 21:27:45.131808 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet adbccf5386624b28af36ff893ac9378f (table rebalancer_extra_table_48 [id=2e1fa2b5e59045abbc77e0d878e997ac]) successfully deleted
I20240502 21:27:45.500870 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:45.505292 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:45.526149 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:45.530329 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:45.565174 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:45.567314 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:45.575913 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:27:45.636070 7328 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 21:27:45.756523 7212 consensus_queue.cc:237] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:45.765285 4838 raft_consensus.cc:1261] T 4cc82d5106804490993007f9f75943aa P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:45.765974 11080 consensus_queue.cc:1035] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:45.769734 7616 raft_consensus.cc:1261] T 4cc82d5106804490993007f9f75943aa P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:45.772423 11080 consensus_queue.cc:1035] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:45.792610 11264 raft_consensus.cc:2932] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:45.800026 7616 raft_consensus.cc:2932] T 4cc82d5106804490993007f9f75943aa P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:45.804749 4298 catalog_manager.cc:5484] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:45.812052 4838 raft_consensus.cc:2932] T 4cc82d5106804490993007f9f75943aa P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:45.839318 7462 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cc82d5106804490993007f9f75943aa with delete_type TABLET_DATA_TOMBSTONED (TS 31a72286397c4edc9a63eca3412be168 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:37182
I20240502 21:27:45.850385 11565 tablet_replica.cc:330] T 4cc82d5106804490993007f9f75943aa P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:27:45.850647 11565 raft_consensus.cc:2229] T 4cc82d5106804490993007f9f75943aa P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:45.850963 11565 raft_consensus.cc:2258] T 4cc82d5106804490993007f9f75943aa P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:45.852159 11565 ts_tablet_manager.cc:1875] T 4cc82d5106804490993007f9f75943aa P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:45.861282 11565 ts_tablet_manager.cc:1888] T 4cc82d5106804490993007f9f75943aa P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:27:45.861845 11565 log.cc:1196] T 4cc82d5106804490993007f9f75943aa P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/4cc82d5106804490993007f9f75943aa
I20240502 21:27:45.864960 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet 4cc82d5106804490993007f9f75943aa (table rebalancer_extra_table_32 [id=9606abae682a4594ada645db2e56a7d4]) successfully deleted
I20240502 21:27:46.195479 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:46.209198 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:46.225482 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:46.250617 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:46.269378 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:46.277514 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:27:46.284060 7328 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 21:27:46.284571 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:46.537483 4440 consensus_queue.cc:237] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:46.539914 4704 raft_consensus.cc:1261] T 81db5e77d00446cba3eb8345ee3df476 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:46.539995 7483 raft_consensus.cc:1261] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:46.540756 11262 consensus_queue.cc:1035] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:46.541070 11153 consensus_queue.cc:1035] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:46.544164 11262 raft_consensus.cc:2932] T 81db5e77d00446cba3eb8345ee3df476 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:46.544889 7483 raft_consensus.cc:2932] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:46.546962 4704 raft_consensus.cc:2932] T 81db5e77d00446cba3eb8345ee3df476 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:46.551612 4295 catalog_manager.cc:5484] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168 reported cstate change: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:46.572517 7194 tablet_service.cc:1514] Processing DeleteTablet for tablet 81db5e77d00446cba3eb8345ee3df476 with delete_type TABLET_DATA_TOMBSTONED (TS 39981a8dfe94462b9a73ba489af4c714 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44652
I20240502 21:27:46.573380 11651 tablet_replica.cc:330] T 81db5e77d00446cba3eb8345ee3df476 P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:27:46.573659 11651 raft_consensus.cc:2229] T 81db5e77d00446cba3eb8345ee3df476 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:46.573933 11651 raft_consensus.cc:2258] T 81db5e77d00446cba3eb8345ee3df476 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:46.574668 11651 ts_tablet_manager.cc:1875] T 81db5e77d00446cba3eb8345ee3df476 P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:46.581959 11651 ts_tablet_manager.cc:1888] T 81db5e77d00446cba3eb8345ee3df476 P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:27:46.582193 11651 log.cc:1196] T 81db5e77d00446cba3eb8345ee3df476 P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/81db5e77d00446cba3eb8345ee3df476
I20240502 21:27:46.583012 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet 81db5e77d00446cba3eb8345ee3df476 (table rebalancer_extra_table_34 [id=e1c7649d4d1248839d2699012aea3ee5]) successfully deleted
I20240502 21:27:46.916113 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:46.918238 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:46.949141 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:46.951843 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:46.997216 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:47.028182 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:47.090409 7328 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 21:27:47.094969 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:27:47.215730 7483 consensus_queue.cc:237] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:47.218026 7215 raft_consensus.cc:1261] T 402e76e6f92744fbb36653d04f861e4f P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:47.218154 4575 raft_consensus.cc:1261] T 402e76e6f92744fbb36653d04f861e4f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:47.218668 10980 consensus_queue.cc:1035] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:47.218978 11223 consensus_queue.cc:1035] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:47.221601 11223 raft_consensus.cc:2932] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:47.223541 4575 raft_consensus.cc:2932] T 402e76e6f92744fbb36653d04f861e4f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:47.232566 7215 raft_consensus.cc:2932] T 402e76e6f92744fbb36653d04f861e4f P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:47.236946 4299 catalog_manager.cc:5484] T 402e76e6f92744fbb36653d04f861e4f P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: config changed from index 2 to 3, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) evicted. New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:47.264057 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet 402e76e6f92744fbb36653d04f861e4f with delete_type TABLET_DATA_TOMBSTONED (TS 71e1375b799b439fa3c83ca763c039c1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:47.264683 11730 tablet_replica.cc:330] T 402e76e6f92744fbb36653d04f861e4f P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:47.264896 11730 raft_consensus.cc:2229] T 402e76e6f92744fbb36653d04f861e4f P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:47.265113 11730 raft_consensus.cc:2258] T 402e76e6f92744fbb36653d04f861e4f P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:47.265754 11730 ts_tablet_manager.cc:1875] T 402e76e6f92744fbb36653d04f861e4f P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:47.276520 11730 ts_tablet_manager.cc:1888] T 402e76e6f92744fbb36653d04f861e4f P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:27:47.278471 11730 log.cc:1196] T 402e76e6f92744fbb36653d04f861e4f P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/402e76e6f92744fbb36653d04f861e4f
I20240502 21:27:47.279387 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet 402e76e6f92744fbb36653d04f861e4f (table rebalancer_extra_table_64 [id=8bc8342ce686477cac74df1ebaa3656a]) successfully deleted
I20240502 21:27:47.619261 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:47.637219 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:47.647763 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:47.689579 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:47.725287 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:47.736420 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:47.765125 7328 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 21:27:47.777817 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:27:47.973793 4575 consensus_queue.cc:237] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:47.975741 7350 raft_consensus.cc:1261] T d6370cfa3d804e0fa5dc2a0f71d78840 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:47.975762 4839 raft_consensus.cc:1261] T d6370cfa3d804e0fa5dc2a0f71d78840 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:47.976403 11691 consensus_queue.cc:1035] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:47.976673 10356 consensus_queue.cc:1035] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:47.980304 10356 raft_consensus.cc:2932] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:47.980823 4839 raft_consensus.cc:2932] T d6370cfa3d804e0fa5dc2a0f71d78840 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:47.984722 7350 raft_consensus.cc:2932] T d6370cfa3d804e0fa5dc2a0f71d78840 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:47.987718 4297 catalog_manager.cc:5484] T d6370cfa3d804e0fa5dc2a0f71d78840 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:48.008394 7462 tablet_service.cc:1514] Processing DeleteTablet for tablet d6370cfa3d804e0fa5dc2a0f71d78840 with delete_type TABLET_DATA_TOMBSTONED (TS 31a72286397c4edc9a63eca3412be168 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:37182
I20240502 21:27:48.019965 11824 tablet_replica.cc:330] T d6370cfa3d804e0fa5dc2a0f71d78840 P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:27:48.020371 11824 raft_consensus.cc:2229] T d6370cfa3d804e0fa5dc2a0f71d78840 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:48.020608 11824 raft_consensus.cc:2258] T d6370cfa3d804e0fa5dc2a0f71d78840 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:48.021214 11824 ts_tablet_manager.cc:1875] T d6370cfa3d804e0fa5dc2a0f71d78840 P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:48.027669 11824 ts_tablet_manager.cc:1888] T d6370cfa3d804e0fa5dc2a0f71d78840 P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:27:48.027858 11824 log.cc:1196] T d6370cfa3d804e0fa5dc2a0f71d78840 P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/d6370cfa3d804e0fa5dc2a0f71d78840
I20240502 21:27:48.028858 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet d6370cfa3d804e0fa5dc2a0f71d78840 (table rebalancer_extra_table_44 [id=0953e644d4b143a9b5a77699e921c8de]) successfully deleted
I20240502 21:27:48.295881 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:48.345350 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:48.400777 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:48.447894 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:48.472373 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:48.549211 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:27:48.585260 7328 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 21:27:48.607722 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:48.689913 7618 consensus_queue.cc:237] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:48.692349 4575 raft_consensus.cc:1261] T 37d864bed6ef46e4a3f57de0373bbe57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:48.692561 4440 raft_consensus.cc:1261] T 37d864bed6ef46e4a3f57de0373bbe57 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:48.692848 11690 consensus_queue.cc:1035] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:48.693120 11861 consensus_queue.cc:1035] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:48.695767 11690 raft_consensus.cc:2932] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:48.696547 4575 raft_consensus.cc:2932] T 37d864bed6ef46e4a3f57de0373bbe57 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:48.699215 4440 raft_consensus.cc:2932] T 37d864bed6ef46e4a3f57de0373bbe57 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:48.702231 4294 catalog_manager.cc:5484] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:48.712152 7328 tablet_service.cc:1514] Processing DeleteTablet for tablet 37d864bed6ef46e4a3f57de0373bbe57 with delete_type TABLET_DATA_TOMBSTONED (TS d0456c5e55a542128688904e003d403c not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47078
I20240502 21:27:48.713393 11907 tablet_replica.cc:330] T 37d864bed6ef46e4a3f57de0373bbe57 P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:27:48.713625 11907 raft_consensus.cc:2229] T 37d864bed6ef46e4a3f57de0373bbe57 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:48.714577 11907 raft_consensus.cc:2258] T 37d864bed6ef46e4a3f57de0373bbe57 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:48.715126 11907 ts_tablet_manager.cc:1875] T 37d864bed6ef46e4a3f57de0373bbe57 P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:48.721210 11907 ts_tablet_manager.cc:1888] T 37d864bed6ef46e4a3f57de0373bbe57 P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:27:48.721331 11907 log.cc:1196] T 37d864bed6ef46e4a3f57de0373bbe57 P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/37d864bed6ef46e4a3f57de0373bbe57
I20240502 21:27:48.722131 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet 37d864bed6ef46e4a3f57de0373bbe57 (table rebalancer_extra_table_36 [id=57ddb6dad3c444239d35a1ed0119bba9]) successfully deleted
I20240502 21:27:49.139750 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:49.155997 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:49.188618 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:49.210116 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:49.265656 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:49.293653 7328 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 21:27:49.297627 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:49.375998 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:27:49.508580 7483 consensus_queue.cc:237] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:49.513651 7348 raft_consensus.cc:1261] T 80304ab48274408393c2e245477c3a78 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:49.513998 4705 raft_consensus.cc:1261] T 80304ab48274408393c2e245477c3a78 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:49.514321 11981 consensus_queue.cc:1035] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:49.514667 11223 consensus_queue.cc:1035] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:49.517555 11981 raft_consensus.cc:2932] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:49.520733 7348 raft_consensus.cc:2932] T 80304ab48274408393c2e245477c3a78 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:49.520747 4705 raft_consensus.cc:2932] T 80304ab48274408393c2e245477c3a78 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:49.528666 4292 catalog_manager.cc:5484] T 80304ab48274408393c2e245477c3a78 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:49.537983 7597 tablet_service.cc:1514] Processing DeleteTablet for tablet 80304ab48274408393c2e245477c3a78 with delete_type TABLET_DATA_TOMBSTONED (TS 0ddd486109f5434cace13d60d633ac04 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44444
I20240502 21:27:49.538549 11991 tablet_replica.cc:330] T 80304ab48274408393c2e245477c3a78 P 0ddd486109f5434cace13d60d633ac04: stopping tablet replica
I20240502 21:27:49.538738 11991 raft_consensus.cc:2229] T 80304ab48274408393c2e245477c3a78 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:49.538923 11991 raft_consensus.cc:2258] T 80304ab48274408393c2e245477c3a78 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:49.539439 11991 ts_tablet_manager.cc:1875] T 80304ab48274408393c2e245477c3a78 P 0ddd486109f5434cace13d60d633ac04: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:49.544013 11991 ts_tablet_manager.cc:1888] T 80304ab48274408393c2e245477c3a78 P 0ddd486109f5434cace13d60d633ac04: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:27:49.544109 11991 log.cc:1196] T 80304ab48274408393c2e245477c3a78 P 0ddd486109f5434cace13d60d633ac04: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/wals/80304ab48274408393c2e245477c3a78
I20240502 21:27:49.544786 4285 catalog_manager.cc:4830] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): tablet 80304ab48274408393c2e245477c3a78 (table rebalancer_extra_table_26 [id=1e6edd8bfe59407fa7863cd780bfae31]) successfully deleted
I20240502 21:27:49.837361 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:49.880733 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:49.915268 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:49.944370 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:49.945181 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:50.047982 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:50.051633 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:27:50.087443 7328 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 21:27:50.214792 4443 consensus_queue.cc:237] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:50.221266 7350 raft_consensus.cc:1261] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:50.221364 4703 raft_consensus.cc:1261] T 1c243b27429043aa8642d9b2ed595ab3 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:50.221820 12075 consensus_queue.cc:1035] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:50.222146 11153 consensus_queue.cc:1035] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:50.225255 12075 raft_consensus.cc:2932] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:50.225844 4703 raft_consensus.cc:2932] T 1c243b27429043aa8642d9b2ed595ab3 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:50.226397 7350 raft_consensus.cc:2932] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:50.231966 4291 catalog_manager.cc:5484] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:50.245759 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c243b27429043aa8642d9b2ed595ab3 with delete_type TABLET_DATA_TOMBSTONED (TS 7a736a8ebb164ce0a051c0fccd76c3b8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60018
I20240502 21:27:50.246379 12081 tablet_replica.cc:330] T 1c243b27429043aa8642d9b2ed595ab3 P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:27:50.246680 12081 raft_consensus.cc:2229] T 1c243b27429043aa8642d9b2ed595ab3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:50.246928 12081 raft_consensus.cc:2258] T 1c243b27429043aa8642d9b2ed595ab3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:50.247757 12081 ts_tablet_manager.cc:1875] T 1c243b27429043aa8642d9b2ed595ab3 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:50.252540 12081 ts_tablet_manager.cc:1888] T 1c243b27429043aa8642d9b2ed595ab3 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:27:50.252647 12081 log.cc:1196] T 1c243b27429043aa8642d9b2ed595ab3 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/1c243b27429043aa8642d9b2ed595ab3
I20240502 21:27:50.253373 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet 1c243b27429043aa8642d9b2ed595ab3 (table rebalance_test_table_17 [id=cc837d2089e24e0fa0a7c398ca3eeca8]) successfully deleted
I20240502 21:27:50.759135 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:50.776845 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:50.784235 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:50.812284 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:50.844491 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:50.859421 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:50.872428 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:27:50.945227 7328 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240502 21:27:51.173547 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.65:42781, user_credentials={real_user=slave}} blocked reactor thread for 46593.2us
I20240502 21:27:51.192886 4575 consensus_queue.cc:237] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:51.195202 7483 raft_consensus.cc:1261] T fed543e42f844227a14a8d812ce158ba P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:27:51.195775 12041 consensus_queue.cc:1035] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:51.196007 4703 raft_consensus.cc:1261] T fed543e42f844227a14a8d812ce158ba P 398c3c8f3195492c900439ad5e1931bb [term 2 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:27:51.196852 12041 consensus_queue.cc:1035] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:51.199124 10356 raft_consensus.cc:2932] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:51.199838 7482 raft_consensus.cc:2932] T fed543e42f844227a14a8d812ce158ba P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:51.200990 4703 raft_consensus.cc:2932] T fed543e42f844227a14a8d812ce158ba P 398c3c8f3195492c900439ad5e1931bb [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:51.207734 4295 catalog_manager.cc:5484] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: config changed from index 2 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New cstate: current_term: 2 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:51.271987 7328 tablet_service.cc:1514] Processing DeleteTablet for tablet fed543e42f844227a14a8d812ce158ba with delete_type TABLET_DATA_TOMBSTONED (TS d0456c5e55a542128688904e003d403c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47078
I20240502 21:27:51.284044 12199 tablet_replica.cc:330] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:27:51.284288 12199 raft_consensus.cc:2229] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:51.284521 12199 raft_consensus.cc:2258] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:51.285197 12199 ts_tablet_manager.cc:1875] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:51.292367 12199 ts_tablet_manager.cc:1888] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240502 21:27:51.295447 12199 log.cc:1196] T fed543e42f844227a14a8d812ce158ba P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/fed543e42f844227a14a8d812ce158ba
I20240502 21:27:51.296375 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet fed543e42f844227a14a8d812ce158ba (table rebalancer_extra_table_56 [id=9ce8c844ead645739f9a918d2d899852]) successfully deleted
I20240502 21:27:51.586498 7616 consensus_queue.cc:237] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:51.589792 4703 raft_consensus.cc:1261] T d09a80f79473469e92310ded0f4edfdd P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:51.589767 7483 raft_consensus.cc:1261] T d09a80f79473469e92310ded0f4edfdd P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:27:51.590153 7549 consensus_peers.cc:485] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d09a80f79473469e92310ded0f4edfdd. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:51.590436 11861 consensus_queue.cc:1035] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:51.590785 12288 consensus_queue.cc:1035] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:51.594485 11690 raft_consensus.cc:2932] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:51.595163 7483 raft_consensus.cc:2932] T d09a80f79473469e92310ded0f4edfdd P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:51.600394 4703 raft_consensus.cc:2932] T d09a80f79473469e92310ded0f4edfdd P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:51.605062 4292 catalog_manager.cc:5484] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 reported cstate change: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:51.646782 7215 consensus_queue.cc:237] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:51.650429 7616 raft_consensus.cc:1261] T a5d47492f64f49d584246fc5007281aa P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:51.651041 12307 consensus_queue.cc:1035] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:27:51.651919 7148 consensus_peers.cc:485] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a5d47492f64f49d584246fc5007281aa. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:51.651522 7483 raft_consensus.cc:1261] T a5d47492f64f49d584246fc5007281aa P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:51.653101 12307 consensus_queue.cc:1035] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:51.656014 11759 raft_consensus.cc:2932] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:51.657442 7483 raft_consensus.cc:2932] T a5d47492f64f49d584246fc5007281aa P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:51.660777 7616 raft_consensus.cc:2932] T a5d47492f64f49d584246fc5007281aa P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:51.666170 4295 catalog_manager.cc:5484] T a5d47492f64f49d584246fc5007281aa P 31a72286397c4edc9a63eca3412be168 reported cstate change: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:51.714844 7348 consensus_queue.cc:237] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:51.717965 7616 raft_consensus.cc:1261] T 180e178d0c714a35b8ca8843fad466c0 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:27:51.718183 7285 consensus_peers.cc:485] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 180e178d0c714a35b8ca8843fad466c0. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:51.718559 12074 consensus_queue.cc:1035] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:51.733574 7483 raft_consensus.cc:1261] T 180e178d0c714a35b8ca8843fad466c0 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:51.734158 12167 consensus_queue.cc:1035] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:51.738760 7483 raft_consensus.cc:2932] T 180e178d0c714a35b8ca8843fad466c0 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:51.738047 12328 raft_consensus.cc:2932] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:51.747027 4292 catalog_manager.cc:5484] T 180e178d0c714a35b8ca8843fad466c0 P 31a72286397c4edc9a63eca3412be168 reported cstate change: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:51.749622 7616 raft_consensus.cc:2932] T 180e178d0c714a35b8ca8843fad466c0 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:51.853104 4705 consensus_queue.cc:237] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:51.856653 4839 raft_consensus.cc:1261] T b7c60d990751440392ab6e5feb0d8498 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:51.856814 4440 raft_consensus.cc:1261] T b7c60d990751440392ab6e5feb0d8498 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:27:51.858264 4641 consensus_peers.cc:485] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb -> Peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Couldn't send request to peer d0456c5e55a542128688904e003d403c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b7c60d990751440392ab6e5feb0d8498. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:51.859087 11265 consensus_queue.cc:1035] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:51.859880 11265 consensus_queue.cc:1035] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:51.882567 12348 raft_consensus.cc:2932] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:51.883267 4839 raft_consensus.cc:2932] T b7c60d990751440392ab6e5feb0d8498 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:51.890897 4443 raft_consensus.cc:2932] T b7c60d990751440392ab6e5feb0d8498 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:51.898746 4292 catalog_manager.cc:5484] T b7c60d990751440392ab6e5feb0d8498 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index -1 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:51.953192 7616 consensus_queue.cc:237] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:51.956568 7483 raft_consensus.cc:1261] T 9e5434cb329e4bea85317d18f6e6867d P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:51.957042 7215 raft_consensus.cc:1261] T 9e5434cb329e4bea85317d18f6e6867d P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:51.957219 12288 consensus_queue.cc:1035] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:51.958040 12288 consensus_queue.cc:1035] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:27:51.958899 7549 consensus_peers.cc:485] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e5434cb329e4bea85317d18f6e6867d. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:51.961565 11690 raft_consensus.cc:2932] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:51.962301 7483 raft_consensus.cc:2932] T 9e5434cb329e4bea85317d18f6e6867d P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:51.967201 7215 raft_consensus.cc:2932] T 9e5434cb329e4bea85317d18f6e6867d P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:51.974790 4295 catalog_manager.cc:5484] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 reported cstate change: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:52.026080 4705 consensus_queue.cc:237] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:52.030620 12387 ts_tablet_manager.cc:902] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328: Initiating tablet copy from peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:52.032298 4838 raft_consensus.cc:1261] T 3a1fadb11cd6436a94e4a67a3aad17ba P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:52.034935 12348 consensus_queue.cc:1035] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:52.044955 4575 raft_consensus.cc:1261] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:52.045691 12351 consensus_queue.cc:1035] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:52.062125 12351 raft_consensus.cc:2932] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:52.069743 12387 tablet_copy_client.cc:308] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.69:33193
I20240502 21:27:52.069350 4575 raft_consensus.cc:2932] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:52.073278 4295 catalog_manager.cc:5484] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb reported cstate change: config changed from index -1 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:52.073267 4838 raft_consensus.cc:2932] T 3a1fadb11cd6436a94e4a67a3aad17ba P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:27:52.075728 7236 tablet_copy_service.cc:140] P 39981a8dfe94462b9a73ba489af4c714: Received BeginTabletCopySession request for tablet a5d47492f64f49d584246fc5007281aa from peer 73a1f2ca1f544de9b0e3b37015f85328 ({username='slave'} at 127.29.82.68:40041)
I20240502 21:27:52.075891 7236 tablet_copy_service.cc:161] P 39981a8dfe94462b9a73ba489af4c714: Beginning new tablet copy session on tablet a5d47492f64f49d584246fc5007281aa from peer 73a1f2ca1f544de9b0e3b37015f85328 at {username='slave'} at 127.29.82.68:40041: session id = 73a1f2ca1f544de9b0e3b37015f85328-a5d47492f64f49d584246fc5007281aa
I20240502 21:27:52.078266 7236 tablet_copy_source_session.cc:205] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:52.082499 12387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5d47492f64f49d584246fc5007281aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:52.088753 12387 tablet_copy_client.cc:783] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:52.089036 12387 tablet_copy_client.cc:647] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:52.094133 12387 tablet_copy_client.cc:523] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:52.098426 12387 tablet_bootstrap.cc:492] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
W20240502 21:27:52.099190 4641 consensus_peers.cc:485] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb -> Peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Couldn't send request to peer 39981a8dfe94462b9a73ba489af4c714. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a1fadb11cd6436a94e4a67a3aad17ba. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:52.107200 12387 tablet_bootstrap.cc:492] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:52.107532 12387 tablet_bootstrap.cc:492] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap complete.
I20240502 21:27:52.107751 12387 ts_tablet_manager.cc:1367] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20240502 21:27:52.108316 12387 raft_consensus.cc:348] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:52.108673 12387 raft_consensus.cc:729] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:52.109047 12387 consensus_queue.cc:260] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:52.115942 12387 ts_tablet_manager.cc:1398] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240502 21:27:52.116772 7236 tablet_copy_service.cc:342] P 39981a8dfe94462b9a73ba489af4c714: Request end of tablet copy session 73a1f2ca1f544de9b0e3b37015f85328-a5d47492f64f49d584246fc5007281aa received from {username='slave'} at 127.29.82.68:40041
I20240502 21:27:52.116904 7236 tablet_copy_service.cc:433] P 39981a8dfe94462b9a73ba489af4c714: ending tablet copy session 73a1f2ca1f544de9b0e3b37015f85328-a5d47492f64f49d584246fc5007281aa on tablet a5d47492f64f49d584246fc5007281aa with peer 73a1f2ca1f544de9b0e3b37015f85328
I20240502 21:27:52.139439 4838 consensus_queue.cc:237] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
W20240502 21:27:52.142180 4773 consensus_peers.cc:485] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a417562f482e4d468095fbe833056233. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:52.142235 7212 raft_consensus.cc:1261] T a417562f482e4d468095fbe833056233 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:52.142740 7616 raft_consensus.cc:1261] T a417562f482e4d468095fbe833056233 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:52.143355 12176 consensus_queue.cc:1035] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:52.143797 9522 consensus_queue.cc:1035] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:52.151414 12176 raft_consensus.cc:2932] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:52.152534 7212 raft_consensus.cc:2932] T a417562f482e4d468095fbe833056233 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:52.158190 7617 raft_consensus.cc:2932] T a417562f482e4d468095fbe833056233 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:52.161609 4295 catalog_manager.cc:5484] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:52.199329 12423 ts_tablet_manager.cc:902] T d09a80f79473469e92310ded0f4edfdd P 71e1375b799b439fa3c83ca763c039c1: Initiating tablet copy from peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:52.206491 12423 tablet_copy_client.cc:308] T d09a80f79473469e92310ded0f4edfdd P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.72:41223
I20240502 21:27:52.206737 12432 ts_tablet_manager.cc:902] T 180e178d0c714a35b8ca8843fad466c0 P 7a736a8ebb164ce0a051c0fccd76c3b8: Initiating tablet copy from peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:52.207625 7638 tablet_copy_service.cc:140] P 0ddd486109f5434cace13d60d633ac04: Received BeginTabletCopySession request for tablet d09a80f79473469e92310ded0f4edfdd from peer 71e1375b799b439fa3c83ca763c039c1 ({username='slave'} at 127.29.82.65:47155)
I20240502 21:27:52.207787 7638 tablet_copy_service.cc:161] P 0ddd486109f5434cace13d60d633ac04: Beginning new tablet copy session on tablet d09a80f79473469e92310ded0f4edfdd from peer 71e1375b799b439fa3c83ca763c039c1 at {username='slave'} at 127.29.82.65:47155: session id = 71e1375b799b439fa3c83ca763c039c1-d09a80f79473469e92310ded0f4edfdd
I20240502 21:27:52.210120 7638 tablet_copy_source_session.cc:205] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:52.211428 12423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d09a80f79473469e92310ded0f4edfdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:52.218322 12423 tablet_copy_client.cc:783] T d09a80f79473469e92310ded0f4edfdd P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:52.219892 12423 tablet_copy_client.cc:647] T d09a80f79473469e92310ded0f4edfdd P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:52.221925 12423 tablet_copy_client.cc:523] T d09a80f79473469e92310ded0f4edfdd P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:52.222283 12432 tablet_copy_client.cc:308] T 180e178d0c714a35b8ca8843fad466c0 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.70:40719
I20240502 21:27:52.224408 7370 tablet_copy_service.cc:140] P d0456c5e55a542128688904e003d403c: Received BeginTabletCopySession request for tablet 180e178d0c714a35b8ca8843fad466c0 from peer 7a736a8ebb164ce0a051c0fccd76c3b8 ({username='slave'} at 127.29.82.66:56529)
I20240502 21:27:52.224555 7370 tablet_copy_service.cc:161] P d0456c5e55a542128688904e003d403c: Beginning new tablet copy session on tablet 180e178d0c714a35b8ca8843fad466c0 from peer 7a736a8ebb164ce0a051c0fccd76c3b8 at {username='slave'} at 127.29.82.66:56529: session id = 7a736a8ebb164ce0a051c0fccd76c3b8-180e178d0c714a35b8ca8843fad466c0
I20240502 21:27:52.225113 12423 tablet_bootstrap.cc:492] T d09a80f79473469e92310ded0f4edfdd P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:52.226747 7370 tablet_copy_source_session.cc:205] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:52.231312 4443 consensus_queue.cc:237] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:52.236218 4705 raft_consensus.cc:1261] T 292bfad92b14456da85bdc874fb7e06e P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:52.236294 4838 raft_consensus.cc:1261] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:27:52.236335 4377 consensus_peers.cc:485] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1 -> Peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Couldn't send request to peer d0456c5e55a542128688904e003d403c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 292bfad92b14456da85bdc874fb7e06e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:52.236856 12206 consensus_queue.cc:1035] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:52.237382 12206 consensus_queue.cc:1035] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:52.244010 12432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 180e178d0c714a35b8ca8843fad466c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:52.264092 12432 tablet_copy_client.cc:783] T 180e178d0c714a35b8ca8843fad466c0 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:52.264398 12432 tablet_copy_client.cc:647] T 180e178d0c714a35b8ca8843fad466c0 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:52.266115 12423 tablet_bootstrap.cc:492] T d09a80f79473469e92310ded0f4edfdd P 71e1375b799b439fa3c83ca763c039c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:52.266513 12423 tablet_bootstrap.cc:492] T d09a80f79473469e92310ded0f4edfdd P 71e1375b799b439fa3c83ca763c039c1: Bootstrap complete.
I20240502 21:27:52.266786 12423 ts_tablet_manager.cc:1367] T d09a80f79473469e92310ded0f4edfdd P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.042s user 0.033s sys 0.008s
I20240502 21:27:52.267606 12423 raft_consensus.cc:348] T d09a80f79473469e92310ded0f4edfdd P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:52.267874 12432 tablet_copy_client.cc:523] T 180e178d0c714a35b8ca8843fad466c0 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:52.267907 12423 raft_consensus.cc:729] T d09a80f79473469e92310ded0f4edfdd P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:52.268219 12423 consensus_queue.cc:260] T d09a80f79473469e92310ded0f4edfdd P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:52.268085 12206 raft_consensus.cc:2932] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:52.268743 12423 ts_tablet_manager.cc:1398] T d09a80f79473469e92310ded0f4edfdd P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:52.270750 7638 tablet_copy_service.cc:342] P 0ddd486109f5434cace13d60d633ac04: Request end of tablet copy session 71e1375b799b439fa3c83ca763c039c1-d09a80f79473469e92310ded0f4edfdd received from {username='slave'} at 127.29.82.65:47155
I20240502 21:27:52.270895 7638 tablet_copy_service.cc:433] P 0ddd486109f5434cace13d60d633ac04: ending tablet copy session 71e1375b799b439fa3c83ca763c039c1-d09a80f79473469e92310ded0f4edfdd on tablet d09a80f79473469e92310ded0f4edfdd with peer 71e1375b799b439fa3c83ca763c039c1
I20240502 21:27:52.271809 12432 tablet_bootstrap.cc:492] T 180e178d0c714a35b8ca8843fad466c0 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:52.278926 4838 raft_consensus.cc:2932] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:52.285372 4292 catalog_manager.cc:5484] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: config changed from index -1 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:52.292932 4705 raft_consensus.cc:2932] T 292bfad92b14456da85bdc874fb7e06e P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:52.318619 12432 tablet_bootstrap.cc:492] T 180e178d0c714a35b8ca8843fad466c0 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:52.319476 12432 tablet_bootstrap.cc:492] T 180e178d0c714a35b8ca8843fad466c0 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap complete.
I20240502 21:27:52.331820 12432 ts_tablet_manager.cc:1367] T 180e178d0c714a35b8ca8843fad466c0 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.060s user 0.007s sys 0.007s
I20240502 21:27:52.332816 12432 raft_consensus.cc:348] T 180e178d0c714a35b8ca8843fad466c0 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:52.333077 12432 raft_consensus.cc:729] T 180e178d0c714a35b8ca8843fad466c0 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:52.333323 12432 consensus_queue.cc:260] T 180e178d0c714a35b8ca8843fad466c0 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:52.334424 12432 ts_tablet_manager.cc:1398] T 180e178d0c714a35b8ca8843fad466c0 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:52.335285 7370 tablet_copy_service.cc:342] P d0456c5e55a542128688904e003d403c: Request end of tablet copy session 7a736a8ebb164ce0a051c0fccd76c3b8-180e178d0c714a35b8ca8843fad466c0 received from {username='slave'} at 127.29.82.66:56529
I20240502 21:27:52.335400 7370 tablet_copy_service.cc:433] P d0456c5e55a542128688904e003d403c: ending tablet copy session 7a736a8ebb164ce0a051c0fccd76c3b8-180e178d0c714a35b8ca8843fad466c0 on tablet 180e178d0c714a35b8ca8843fad466c0 with peer 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:27:52.380674 4443 consensus_queue.cc:237] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:52.383831 4572 raft_consensus.cc:1261] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:52.383834 7215 raft_consensus.cc:1261] T 2eb73e73b64b4b0f9e5116f258f2e147 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:52.384402 12075 consensus_queue.cc:1035] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:27:52.385385 4375 consensus_peers.cc:485] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1 -> Peer 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): Couldn't send request to peer 31a72286397c4edc9a63eca3412be168. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2eb73e73b64b4b0f9e5116f258f2e147. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:52.386289 12075 consensus_queue.cc:1035] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:52.407631 12075 raft_consensus.cc:2932] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:27:52.413354 7212 raft_consensus.cc:2932] T 2eb73e73b64b4b0f9e5116f258f2e147 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:27:52.414438 4575 raft_consensus.cc:2932] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:27:52.415935 4292 catalog_manager.cc:5484] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: config changed from index -1 to 2, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:27:52.459954 12483 ts_tablet_manager.cc:902] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c: Initiating tablet copy from peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:27:52.461225 12483 tablet_copy_client.cc:308] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.67:34537
I20240502 21:27:52.461894 4727 tablet_copy_service.cc:140] P 398c3c8f3195492c900439ad5e1931bb: Received BeginTabletCopySession request for tablet b7c60d990751440392ab6e5feb0d8498 from peer d0456c5e55a542128688904e003d403c ({username='slave'} at 127.29.82.70:43047)
I20240502 21:27:52.462029 4727 tablet_copy_service.cc:161] P 398c3c8f3195492c900439ad5e1931bb: Beginning new tablet copy session on tablet b7c60d990751440392ab6e5feb0d8498 from peer d0456c5e55a542128688904e003d403c at {username='slave'} at 127.29.82.70:43047: session id = d0456c5e55a542128688904e003d403c-b7c60d990751440392ab6e5feb0d8498
I20240502 21:27:52.464668 4727 tablet_copy_source_session.cc:205] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:52.465271 7348 consensus_queue.cc:237] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:52.465864 12483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7c60d990751440392ab6e5feb0d8498. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:52.468384 4839 raft_consensus.cc:1261] T 84c8bdb300064adc86a4b23e07d512d3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:52.468390 4575 raft_consensus.cc:1261] T 84c8bdb300064adc86a4b23e07d512d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:27:52.469055 7281 consensus_peers.cc:485] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84c8bdb300064adc86a4b23e07d512d3. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:52.469189 12167 consensus_queue.cc:1035] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:52.470575 12167 consensus_queue.cc:1035] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:52.473269 12167 raft_consensus.cc:2932] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:52.474356 4575 raft_consensus.cc:2932] T 84c8bdb300064adc86a4b23e07d512d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:52.476454 12483 tablet_copy_client.cc:783] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:52.476645 12483 tablet_copy_client.cc:647] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:52.477988 4839 raft_consensus.cc:2932] T 84c8bdb300064adc86a4b23e07d512d3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:52.478442 4838 raft_consensus.cc:1203] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:52.485884 4295 catalog_manager.cc:5484] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c reported cstate change: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:27:52.488757 12483 tablet_copy_client.cc:523] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:52.492445 12483 tablet_bootstrap.cc:492] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:52.540966 12483 tablet_bootstrap.cc:492] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:52.541393 12483 tablet_bootstrap.cc:492] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c: Bootstrap complete.
I20240502 21:27:52.541677 12483 ts_tablet_manager.cc:1367] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.049s user 0.044s sys 0.000s
I20240502 21:27:52.542320 12483 raft_consensus.cc:348] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:52.542490 12483 raft_consensus.cc:729] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:52.542637 12483 consensus_queue.cc:260] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:52.543169 12483 ts_tablet_manager.cc:1398] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:27:52.543942 4727 tablet_copy_service.cc:342] P 398c3c8f3195492c900439ad5e1931bb: Request end of tablet copy session d0456c5e55a542128688904e003d403c-b7c60d990751440392ab6e5feb0d8498 received from {username='slave'} at 127.29.82.70:43047
I20240502 21:27:52.544073 4727 tablet_copy_service.cc:433] P 398c3c8f3195492c900439ad5e1931bb: ending tablet copy session d0456c5e55a542128688904e003d403c-b7c60d990751440392ab6e5feb0d8498 on tablet b7c60d990751440392ab6e5feb0d8498 with peer d0456c5e55a542128688904e003d403c
I20240502 21:27:52.547995 7616 consensus_queue.cc:237] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:52.548636 12423 ts_tablet_manager.cc:902] T 9e5434cb329e4bea85317d18f6e6867d P 71e1375b799b439fa3c83ca763c039c1: Initiating tablet copy from peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:52.549904 12423 tablet_copy_client.cc:308] T 9e5434cb329e4bea85317d18f6e6867d P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.72:41223
I20240502 21:27:52.551196 7638 tablet_copy_service.cc:140] P 0ddd486109f5434cace13d60d633ac04: Received BeginTabletCopySession request for tablet 9e5434cb329e4bea85317d18f6e6867d from peer 71e1375b799b439fa3c83ca763c039c1 ({username='slave'} at 127.29.82.65:47155)
I20240502 21:27:52.551337 7638 tablet_copy_service.cc:161] P 0ddd486109f5434cace13d60d633ac04: Beginning new tablet copy session on tablet 9e5434cb329e4bea85317d18f6e6867d from peer 71e1375b799b439fa3c83ca763c039c1 at {username='slave'} at 127.29.82.65:47155: session id = 71e1375b799b439fa3c83ca763c039c1-9e5434cb329e4bea85317d18f6e6867d
W20240502 21:27:52.552109 7552 consensus_peers.cc:485] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 29235795acc147f68d2ccce93564ebd3. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:52.552153 4705 raft_consensus.cc:1261] T 29235795acc147f68d2ccce93564ebd3 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:52.552850 7483 raft_consensus.cc:1261] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:52.553398 11861 consensus_queue.cc:1035] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:52.553764 7638 tablet_copy_source_session.cc:205] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:52.554042 11861 consensus_queue.cc:1035] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:52.555039 12423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e5434cb329e4bea85317d18f6e6867d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:52.557914 11690 raft_consensus.cc:2932] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:52.560253 12423 tablet_copy_client.cc:783] T 9e5434cb329e4bea85317d18f6e6867d P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:52.560470 12423 tablet_copy_client.cc:647] T 9e5434cb329e4bea85317d18f6e6867d P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:52.560726 4705 raft_consensus.cc:2932] T 29235795acc147f68d2ccce93564ebd3 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:52.563858 12423 tablet_copy_client.cc:523] T 9e5434cb329e4bea85317d18f6e6867d P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:52.564407 12432 ts_tablet_manager.cc:902] T a417562f482e4d468095fbe833056233 P 7a736a8ebb164ce0a051c0fccd76c3b8: Initiating tablet copy from peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:52.565780 12432 tablet_copy_client.cc:308] T a417562f482e4d468095fbe833056233 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.68:42449
I20240502 21:27:52.566478 4859 tablet_copy_service.cc:140] P 73a1f2ca1f544de9b0e3b37015f85328: Received BeginTabletCopySession request for tablet a417562f482e4d468095fbe833056233 from peer 7a736a8ebb164ce0a051c0fccd76c3b8 ({username='slave'} at 127.29.82.66:35827)
I20240502 21:27:52.566635 4859 tablet_copy_service.cc:161] P 73a1f2ca1f544de9b0e3b37015f85328: Beginning new tablet copy session on tablet a417562f482e4d468095fbe833056233 from peer 7a736a8ebb164ce0a051c0fccd76c3b8 at {username='slave'} at 127.29.82.66:35827: session id = 7a736a8ebb164ce0a051c0fccd76c3b8-a417562f482e4d468095fbe833056233
I20240502 21:27:52.568838 12423 tablet_bootstrap.cc:492] T 9e5434cb329e4bea85317d18f6e6867d P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:52.569324 4859 tablet_copy_source_session.cc:205] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:52.570645 12432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a417562f482e4d468095fbe833056233. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:52.579653 12432 tablet_copy_client.cc:783] T a417562f482e4d468095fbe833056233 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:52.579900 12432 tablet_copy_client.cc:647] T a417562f482e4d468095fbe833056233 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:52.580991 4292 catalog_manager.cc:5484] T 29235795acc147f68d2ccce93564ebd3 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:52.583669 12432 tablet_copy_client.cc:523] T a417562f482e4d468095fbe833056233 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:52.586441 12423 tablet_bootstrap.cc:492] T 9e5434cb329e4bea85317d18f6e6867d P 71e1375b799b439fa3c83ca763c039c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:52.587368 12432 tablet_bootstrap.cc:492] T a417562f482e4d468095fbe833056233 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:52.588402 12423 tablet_bootstrap.cc:492] T 9e5434cb329e4bea85317d18f6e6867d P 71e1375b799b439fa3c83ca763c039c1: Bootstrap complete.
I20240502 21:27:52.588696 12423 ts_tablet_manager.cc:1367] T 9e5434cb329e4bea85317d18f6e6867d P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.000s
I20240502 21:27:52.590374 12423 raft_consensus.cc:348] T 9e5434cb329e4bea85317d18f6e6867d P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:52.590626 12423 raft_consensus.cc:729] T 9e5434cb329e4bea85317d18f6e6867d P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:52.591697 12423 consensus_queue.cc:260] T 9e5434cb329e4bea85317d18f6e6867d P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:52.595975 12423 ts_tablet_manager.cc:1398] T 9e5434cb329e4bea85317d18f6e6867d P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 21:27:52.597910 12432 tablet_bootstrap.cc:492] T a417562f482e4d468095fbe833056233 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:52.598613 12432 tablet_bootstrap.cc:492] T a417562f482e4d468095fbe833056233 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap complete.
I20240502 21:27:52.598851 12432 ts_tablet_manager.cc:1367] T a417562f482e4d468095fbe833056233 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20240502 21:27:52.599402 7638 tablet_copy_service.cc:342] P 0ddd486109f5434cace13d60d633ac04: Request end of tablet copy session 71e1375b799b439fa3c83ca763c039c1-9e5434cb329e4bea85317d18f6e6867d received from {username='slave'} at 127.29.82.65:47155
I20240502 21:27:52.599514 7638 tablet_copy_service.cc:433] P 0ddd486109f5434cace13d60d633ac04: ending tablet copy session 71e1375b799b439fa3c83ca763c039c1-9e5434cb329e4bea85317d18f6e6867d on tablet 9e5434cb329e4bea85317d18f6e6867d with peer 71e1375b799b439fa3c83ca763c039c1
I20240502 21:27:52.600579 12432 raft_consensus.cc:348] T a417562f482e4d468095fbe833056233 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:52.600787 12432 raft_consensus.cc:729] T a417562f482e4d468095fbe833056233 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:52.600987 12432 consensus_queue.cc:260] T a417562f482e4d468095fbe833056233 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:52.601712 12432 ts_tablet_manager.cc:1398] T a417562f482e4d468095fbe833056233 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:52.602794 4859 tablet_copy_service.cc:342] P 73a1f2ca1f544de9b0e3b37015f85328: Request end of tablet copy session 7a736a8ebb164ce0a051c0fccd76c3b8-a417562f482e4d468095fbe833056233 received from {username='slave'} at 127.29.82.66:35827
I20240502 21:27:52.602916 4859 tablet_copy_service.cc:433] P 73a1f2ca1f544de9b0e3b37015f85328: ending tablet copy session 7a736a8ebb164ce0a051c0fccd76c3b8-a417562f482e4d468095fbe833056233 on tablet a417562f482e4d468095fbe833056233 with peer 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:27:52.607071 7483 raft_consensus.cc:2932] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:52.636348 12483 ts_tablet_manager.cc:902] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c: Initiating tablet copy from peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:52.639971 12483 tablet_copy_client.cc:308] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.65:42781
I20240502 21:27:52.640566 4463 tablet_copy_service.cc:140] P 71e1375b799b439fa3c83ca763c039c1: Received BeginTabletCopySession request for tablet 292bfad92b14456da85bdc874fb7e06e from peer d0456c5e55a542128688904e003d403c ({username='slave'} at 127.29.82.70:41689)
I20240502 21:27:52.640725 4463 tablet_copy_service.cc:161] P 71e1375b799b439fa3c83ca763c039c1: Beginning new tablet copy session on tablet 292bfad92b14456da85bdc874fb7e06e from peer d0456c5e55a542128688904e003d403c at {username='slave'} at 127.29.82.70:41689: session id = d0456c5e55a542128688904e003d403c-292bfad92b14456da85bdc874fb7e06e
I20240502 21:27:52.643620 4463 tablet_copy_source_session.cc:205] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:52.644800 12483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 292bfad92b14456da85bdc874fb7e06e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:52.650213 12483 tablet_copy_client.cc:783] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:52.650431 12483 tablet_copy_client.cc:647] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:52.652171 12483 tablet_copy_client.cc:523] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:52.655453 12483 tablet_bootstrap.cc:492] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:52.668330 12544 ts_tablet_manager.cc:902] T 3a1fadb11cd6436a94e4a67a3aad17ba P 39981a8dfe94462b9a73ba489af4c714: Initiating tablet copy from peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:27:52.669215 4839 consensus_queue.cc:237] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:52.674120 4705 raft_consensus.cc:1261] T dd1cf8e447e54d6e907ffa0e4681515f P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:52.674165 4572 raft_consensus.cc:1261] T dd1cf8e447e54d6e907ffa0e4681515f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:52.674962 12176 consensus_queue.cc:1035] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:52.675206 9522 consensus_queue.cc:1035] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:27:52.675606 4772 consensus_peers.cc:485] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Couldn't send request to peer d0456c5e55a542128688904e003d403c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd1cf8e447e54d6e907ffa0e4681515f. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:52.691210 9522 raft_consensus.cc:2932] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:52.695767 12544 tablet_copy_client.cc:308] T 3a1fadb11cd6436a94e4a67a3aad17ba P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.67:34537
I20240502 21:27:52.696477 4727 tablet_copy_service.cc:140] P 398c3c8f3195492c900439ad5e1931bb: Received BeginTabletCopySession request for tablet 3a1fadb11cd6436a94e4a67a3aad17ba from peer 39981a8dfe94462b9a73ba489af4c714 ({username='slave'} at 127.29.82.69:46901)
I20240502 21:27:52.696650 4727 tablet_copy_service.cc:161] P 398c3c8f3195492c900439ad5e1931bb: Beginning new tablet copy session on tablet 3a1fadb11cd6436a94e4a67a3aad17ba from peer 39981a8dfe94462b9a73ba489af4c714 at {username='slave'} at 127.29.82.69:46901: session id = 39981a8dfe94462b9a73ba489af4c714-3a1fadb11cd6436a94e4a67a3aad17ba
I20240502 21:27:52.698083 4705 raft_consensus.cc:2932] T dd1cf8e447e54d6e907ffa0e4681515f P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:52.699453 4727 tablet_copy_source_session.cc:205] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:52.702910 4300 catalog_manager.cc:5484] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index -1 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:52.704710 12544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a1fadb11cd6436a94e4a67a3aad17ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:52.707970 4572 raft_consensus.cc:2932] T dd1cf8e447e54d6e907ffa0e4681515f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:52.711563 12544 tablet_copy_client.cc:783] T 3a1fadb11cd6436a94e4a67a3aad17ba P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:52.712015 12544 tablet_copy_client.cc:647] T 3a1fadb11cd6436a94e4a67a3aad17ba P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:52.723075 12483 tablet_bootstrap.cc:492] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:52.723410 12483 tablet_bootstrap.cc:492] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c: Bootstrap complete.
I20240502 21:27:52.723681 12483 ts_tablet_manager.cc:1367] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.068s user 0.033s sys 0.011s
I20240502 21:27:52.724499 12483 raft_consensus.cc:348] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:52.724699 12483 raft_consensus.cc:729] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:52.724952 12483 consensus_queue.cc:260] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:52.725327 12544 tablet_copy_client.cc:523] T 3a1fadb11cd6436a94e4a67a3aad17ba P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:52.725827 12483 ts_tablet_manager.cc:1398] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:52.726519 4463 tablet_copy_service.cc:342] P 71e1375b799b439fa3c83ca763c039c1: Request end of tablet copy session d0456c5e55a542128688904e003d403c-292bfad92b14456da85bdc874fb7e06e received from {username='slave'} at 127.29.82.70:41689
I20240502 21:27:52.726637 4463 tablet_copy_service.cc:433] P 71e1375b799b439fa3c83ca763c039c1: ending tablet copy session d0456c5e55a542128688904e003d403c-292bfad92b14456da85bdc874fb7e06e on tablet 292bfad92b14456da85bdc874fb7e06e with peer d0456c5e55a542128688904e003d403c
I20240502 21:27:52.731290 12544 tablet_bootstrap.cc:492] T 3a1fadb11cd6436a94e4a67a3aad17ba P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:27:52.753461 4440 raft_consensus.cc:1203] T d09a80f79473469e92310ded0f4edfdd P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:52.817826 4575 raft_consensus.cc:1203] T 180e178d0c714a35b8ca8843fad466c0 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:52.856531 12544 tablet_bootstrap.cc:492] T 3a1fadb11cd6436a94e4a67a3aad17ba P 39981a8dfe94462b9a73ba489af4c714: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:52.857555 12544 tablet_bootstrap.cc:492] T 3a1fadb11cd6436a94e4a67a3aad17ba P 39981a8dfe94462b9a73ba489af4c714: Bootstrap complete.
I20240502 21:27:52.857909 12544 ts_tablet_manager.cc:1367] T 3a1fadb11cd6436a94e4a67a3aad17ba P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.127s user 0.056s sys 0.005s
I20240502 21:27:52.858739 12544 raft_consensus.cc:348] T 3a1fadb11cd6436a94e4a67a3aad17ba P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:52.859720 12544 raft_consensus.cc:729] T 3a1fadb11cd6436a94e4a67a3aad17ba P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:27:52.859980 12544 consensus_queue.cc:260] T 3a1fadb11cd6436a94e4a67a3aad17ba P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:27:52.865633 12544 ts_tablet_manager.cc:1398] T 3a1fadb11cd6436a94e4a67a3aad17ba P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 21:27:52.866348 4727 tablet_copy_service.cc:342] P 398c3c8f3195492c900439ad5e1931bb: Request end of tablet copy session 39981a8dfe94462b9a73ba489af4c714-3a1fadb11cd6436a94e4a67a3aad17ba received from {username='slave'} at 127.29.82.69:46901
I20240502 21:27:52.866468 4727 tablet_copy_service.cc:433] P 398c3c8f3195492c900439ad5e1931bb: ending tablet copy session 39981a8dfe94462b9a73ba489af4c714-3a1fadb11cd6436a94e4a67a3aad17ba on tablet 3a1fadb11cd6436a94e4a67a3aad17ba with peer 39981a8dfe94462b9a73ba489af4c714
I20240502 21:27:52.876693 12563 ts_tablet_manager.cc:902] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168: Initiating tablet copy from peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:52.894730 4440 consensus_queue.cc:237] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
W20240502 21:27:52.905566 4376 consensus_peers.cc:485] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1 -> Peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): Couldn't send request to peer 0ddd486109f5434cace13d60d633ac04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ca332df7aaef4ddd9fd3266e8003b55f. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:52.906009 7350 raft_consensus.cc:1261] T ca332df7aaef4ddd9fd3266e8003b55f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:52.907267 7212 raft_consensus.cc:1261] T ca332df7aaef4ddd9fd3266e8003b55f P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:52.908067 12075 consensus_queue.cc:1035] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:52.908783 12075 consensus_queue.cc:1035] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:52.935262 12464 raft_consensus.cc:2932] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:52.936002 12563 tablet_copy_client.cc:308] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.65:42781
I20240502 21:27:52.939335 4463 tablet_copy_service.cc:140] P 71e1375b799b439fa3c83ca763c039c1: Received BeginTabletCopySession request for tablet 2eb73e73b64b4b0f9e5116f258f2e147 from peer 31a72286397c4edc9a63eca3412be168 ({username='slave'} at 127.29.82.71:41363)
I20240502 21:27:52.939477 4463 tablet_copy_service.cc:161] P 71e1375b799b439fa3c83ca763c039c1: Beginning new tablet copy session on tablet 2eb73e73b64b4b0f9e5116f258f2e147 from peer 31a72286397c4edc9a63eca3412be168 at {username='slave'} at 127.29.82.71:41363: session id = 31a72286397c4edc9a63eca3412be168-2eb73e73b64b4b0f9e5116f258f2e147
I20240502 21:27:52.939455 7350 raft_consensus.cc:1203] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:52.939421 7349 raft_consensus.cc:2932] T ca332df7aaef4ddd9fd3266e8003b55f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:52.941560 4463 tablet_copy_source_session.cc:205] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:52.944304 12563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2eb73e73b64b4b0f9e5116f258f2e147. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:52.945874 4300 catalog_manager.cc:5484] T ca332df7aaef4ddd9fd3266e8003b55f P d0456c5e55a542128688904e003d403c reported cstate change: config changed from index -1 to 2, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:52.951301 12563 tablet_copy_client.cc:783] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:52.951684 12563 tablet_copy_client.cc:647] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:52.952471 7212 raft_consensus.cc:2932] T ca332df7aaef4ddd9fd3266e8003b55f P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:52.954280 12563 tablet_copy_client.cc:523] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:52.958681 12563 tablet_bootstrap.cc:492] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:27:52.966264 12563 tablet_bootstrap.cc:492] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:52.966862 12563 tablet_bootstrap.cc:492] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168: Bootstrap complete.
I20240502 21:27:52.967221 12563 ts_tablet_manager.cc:1367] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240502 21:27:52.968273 12563 raft_consensus.cc:348] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:52.968729 12563 raft_consensus.cc:729] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:27:52.969201 12563 consensus_queue.cc:260] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:52.970223 12563 ts_tablet_manager.cc:1398] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:52.971189 4463 tablet_copy_service.cc:342] P 71e1375b799b439fa3c83ca763c039c1: Request end of tablet copy session 31a72286397c4edc9a63eca3412be168-2eb73e73b64b4b0f9e5116f258f2e147 received from {username='slave'} at 127.29.82.71:41363
I20240502 21:27:52.971320 4463 tablet_copy_service.cc:433] P 71e1375b799b439fa3c83ca763c039c1: ending tablet copy session 31a72286397c4edc9a63eca3412be168-2eb73e73b64b4b0f9e5116f258f2e147 on tablet 2eb73e73b64b4b0f9e5116f258f2e147 with peer 31a72286397c4edc9a63eca3412be168
I20240502 21:27:53.020720 12423 ts_tablet_manager.cc:902] T 84c8bdb300064adc86a4b23e07d512d3 P 71e1375b799b439fa3c83ca763c039c1: Initiating tablet copy from peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:53.044376 7212 raft_consensus.cc:1203] T 3a1fadb11cd6436a94e4a67a3aad17ba P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:53.055855 4440 raft_consensus.cc:1203] T 9e5434cb329e4bea85317d18f6e6867d P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:53.061618 12423 tablet_copy_client.cc:308] T 84c8bdb300064adc86a4b23e07d512d3 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.70:40719
I20240502 21:27:53.062326 7370 tablet_copy_service.cc:140] P d0456c5e55a542128688904e003d403c: Received BeginTabletCopySession request for tablet 84c8bdb300064adc86a4b23e07d512d3 from peer 71e1375b799b439fa3c83ca763c039c1 ({username='slave'} at 127.29.82.65:53175)
I20240502 21:27:53.062506 7370 tablet_copy_service.cc:161] P d0456c5e55a542128688904e003d403c: Beginning new tablet copy session on tablet 84c8bdb300064adc86a4b23e07d512d3 from peer 71e1375b799b439fa3c83ca763c039c1 at {username='slave'} at 127.29.82.65:53175: session id = 71e1375b799b439fa3c83ca763c039c1-84c8bdb300064adc86a4b23e07d512d3
I20240502 21:27:53.065429 7370 tablet_copy_source_session.cc:205] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:53.068980 12423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84c8bdb300064adc86a4b23e07d512d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:53.071612 7617 consensus_queue.cc:237] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:53.074373 7482 raft_consensus.cc:1261] T 8991236419e642bfa438b9e45a1b8235 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:53.074837 12423 tablet_copy_client.cc:783] T 84c8bdb300064adc86a4b23e07d512d3 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:53.074864 7349 raft_consensus.cc:1261] T 8991236419e642bfa438b9e45a1b8235 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:27:53.074913 7552 consensus_peers.cc:485] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8991236419e642bfa438b9e45a1b8235. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:53.075042 12423 tablet_copy_client.cc:647] T 84c8bdb300064adc86a4b23e07d512d3 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:53.075414 12523 consensus_queue.cc:1035] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:53.075794 11861 consensus_queue.cc:1035] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:53.077196 12423 tablet_copy_client.cc:523] T 84c8bdb300064adc86a4b23e07d512d3 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:53.079173 12523 raft_consensus.cc:2932] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:53.079794 7482 raft_consensus.cc:2932] T 8991236419e642bfa438b9e45a1b8235 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:53.080588 12423 tablet_bootstrap.cc:492] T 84c8bdb300064adc86a4b23e07d512d3 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:27:53.084643 7349 raft_consensus.cc:2932] T 8991236419e642bfa438b9e45a1b8235 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:53.089285 4292 catalog_manager.cc:5484] T 8991236419e642bfa438b9e45a1b8235 P 31a72286397c4edc9a63eca3412be168 reported cstate change: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:53.109344 12432 ts_tablet_manager.cc:902] T 29235795acc147f68d2ccce93564ebd3 P 7a736a8ebb164ce0a051c0fccd76c3b8: Initiating tablet copy from peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:53.114207 12423 tablet_bootstrap.cc:492] T 84c8bdb300064adc86a4b23e07d512d3 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:53.114745 12423 tablet_bootstrap.cc:492] T 84c8bdb300064adc86a4b23e07d512d3 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap complete.
I20240502 21:27:53.115065 12423 ts_tablet_manager.cc:1367] T 84c8bdb300064adc86a4b23e07d512d3 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.016s
I20240502 21:27:53.116953 4575 raft_consensus.cc:1203] T a417562f482e4d468095fbe833056233 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:53.117895 12423 raft_consensus.cc:348] T 84c8bdb300064adc86a4b23e07d512d3 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:53.118248 12423 raft_consensus.cc:729] T 84c8bdb300064adc86a4b23e07d512d3 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:27:53.118554 12423 consensus_queue.cc:260] T 84c8bdb300064adc86a4b23e07d512d3 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:27:53.122484 12423 ts_tablet_manager.cc:1398] T 84c8bdb300064adc86a4b23e07d512d3 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 21:27:53.123260 7370 tablet_copy_service.cc:342] P d0456c5e55a542128688904e003d403c: Request end of tablet copy session 71e1375b799b439fa3c83ca763c039c1-84c8bdb300064adc86a4b23e07d512d3 received from {username='slave'} at 127.29.82.65:53175
I20240502 21:27:53.123383 7370 tablet_copy_service.cc:433] P d0456c5e55a542128688904e003d403c: ending tablet copy session 71e1375b799b439fa3c83ca763c039c1-84c8bdb300064adc86a4b23e07d512d3 on tablet 84c8bdb300064adc86a4b23e07d512d3 with peer 71e1375b799b439fa3c83ca763c039c1
I20240502 21:27:53.129622 12483 ts_tablet_manager.cc:902] T dd1cf8e447e54d6e907ffa0e4681515f P d0456c5e55a542128688904e003d403c: Initiating tablet copy from peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:27:53.132453 12483 tablet_copy_client.cc:308] T dd1cf8e447e54d6e907ffa0e4681515f P d0456c5e55a542128688904e003d403c: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.68:42449
I20240502 21:27:53.133212 4859 tablet_copy_service.cc:140] P 73a1f2ca1f544de9b0e3b37015f85328: Received BeginTabletCopySession request for tablet dd1cf8e447e54d6e907ffa0e4681515f from peer d0456c5e55a542128688904e003d403c ({username='slave'} at 127.29.82.70:52223)
I20240502 21:27:53.133378 4859 tablet_copy_service.cc:161] P 73a1f2ca1f544de9b0e3b37015f85328: Beginning new tablet copy session on tablet dd1cf8e447e54d6e907ffa0e4681515f from peer d0456c5e55a542128688904e003d403c at {username='slave'} at 127.29.82.70:52223: session id = d0456c5e55a542128688904e003d403c-dd1cf8e447e54d6e907ffa0e4681515f
I20240502 21:27:53.135950 4859 tablet_copy_source_session.cc:205] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:53.137109 12483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd1cf8e447e54d6e907ffa0e4681515f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:53.142115 12483 tablet_copy_client.cc:783] T dd1cf8e447e54d6e907ffa0e4681515f P d0456c5e55a542128688904e003d403c: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:53.142692 12483 tablet_copy_client.cc:647] T dd1cf8e447e54d6e907ffa0e4681515f P d0456c5e55a542128688904e003d403c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:53.144932 7349 raft_consensus.cc:1203] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:53.158970 12483 tablet_copy_client.cc:523] T dd1cf8e447e54d6e907ffa0e4681515f P d0456c5e55a542128688904e003d403c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:53.166894 12483 tablet_bootstrap.cc:492] T dd1cf8e447e54d6e907ffa0e4681515f P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:27:53.200311 12432 tablet_copy_client.cc:308] T 29235795acc147f68d2ccce93564ebd3 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.72:41223
I20240502 21:27:53.201025 7638 tablet_copy_service.cc:140] P 0ddd486109f5434cace13d60d633ac04: Received BeginTabletCopySession request for tablet 29235795acc147f68d2ccce93564ebd3 from peer 7a736a8ebb164ce0a051c0fccd76c3b8 ({username='slave'} at 127.29.82.66:56819)
I20240502 21:27:53.201176 7638 tablet_copy_service.cc:161] P 0ddd486109f5434cace13d60d633ac04: Beginning new tablet copy session on tablet 29235795acc147f68d2ccce93564ebd3 from peer 7a736a8ebb164ce0a051c0fccd76c3b8 at {username='slave'} at 127.29.82.66:56819: session id = 7a736a8ebb164ce0a051c0fccd76c3b8-29235795acc147f68d2ccce93564ebd3
I20240502 21:27:53.203346 7638 tablet_copy_source_session.cc:205] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:53.204511 12432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29235795acc147f68d2ccce93564ebd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:53.209863 12432 tablet_copy_client.cc:783] T 29235795acc147f68d2ccce93564ebd3 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:53.210103 12432 tablet_copy_client.cc:647] T 29235795acc147f68d2ccce93564ebd3 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:53.211640 12432 tablet_copy_client.cc:523] T 29235795acc147f68d2ccce93564ebd3 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:53.214407 12432 tablet_bootstrap.cc:492] T 29235795acc147f68d2ccce93564ebd3 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:53.227758 12432 tablet_bootstrap.cc:492] T 29235795acc147f68d2ccce93564ebd3 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:53.228210 12432 tablet_bootstrap.cc:492] T 29235795acc147f68d2ccce93564ebd3 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap complete.
I20240502 21:27:53.228518 12432 ts_tablet_manager.cc:1367] T 29235795acc147f68d2ccce93564ebd3 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.004s
I20240502 21:27:53.229419 12432 raft_consensus.cc:348] T 29235795acc147f68d2ccce93564ebd3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:53.229846 12432 raft_consensus.cc:729] T 29235795acc147f68d2ccce93564ebd3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:53.230106 12432 consensus_queue.cc:260] T 29235795acc147f68d2ccce93564ebd3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:53.230908 12432 ts_tablet_manager.cc:1398] T 29235795acc147f68d2ccce93564ebd3 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:53.232197 7638 tablet_copy_service.cc:342] P 0ddd486109f5434cace13d60d633ac04: Request end of tablet copy session 7a736a8ebb164ce0a051c0fccd76c3b8-29235795acc147f68d2ccce93564ebd3 received from {username='slave'} at 127.29.82.66:56819
I20240502 21:27:53.232319 7638 tablet_copy_service.cc:433] P 0ddd486109f5434cace13d60d633ac04: ending tablet copy session 7a736a8ebb164ce0a051c0fccd76c3b8-29235795acc147f68d2ccce93564ebd3 on tablet 29235795acc147f68d2ccce93564ebd3 with peer 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:27:53.245930 12483 tablet_bootstrap.cc:492] T dd1cf8e447e54d6e907ffa0e4681515f P d0456c5e55a542128688904e003d403c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:53.246380 12483 tablet_bootstrap.cc:492] T dd1cf8e447e54d6e907ffa0e4681515f P d0456c5e55a542128688904e003d403c: Bootstrap complete.
I20240502 21:27:53.246687 12483 ts_tablet_manager.cc:1367] T dd1cf8e447e54d6e907ffa0e4681515f P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.080s user 0.048s sys 0.004s
I20240502 21:27:53.247532 12483 raft_consensus.cc:348] T dd1cf8e447e54d6e907ffa0e4681515f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:53.247825 12483 raft_consensus.cc:729] T dd1cf8e447e54d6e907ffa0e4681515f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:27:53.248123 12483 consensus_queue.cc:260] T dd1cf8e447e54d6e907ffa0e4681515f P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:53.248847 12483 ts_tablet_manager.cc:1398] T dd1cf8e447e54d6e907ffa0e4681515f P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240502 21:27:53.249826 4859 tablet_copy_service.cc:342] P 73a1f2ca1f544de9b0e3b37015f85328: Request end of tablet copy session d0456c5e55a542128688904e003d403c-dd1cf8e447e54d6e907ffa0e4681515f received from {username='slave'} at 127.29.82.70:52223
I20240502 21:27:53.249948 4859 tablet_copy_service.cc:433] P 73a1f2ca1f544de9b0e3b37015f85328: ending tablet copy session d0456c5e55a542128688904e003d403c-dd1cf8e447e54d6e907ffa0e4681515f on tablet dd1cf8e447e54d6e907ffa0e4681515f with peer d0456c5e55a542128688904e003d403c
I20240502 21:27:53.290057 7482 consensus_queue.cc:237] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:53.296094 7349 raft_consensus.cc:1261] T b4298fc9f7694af19716ef4a908a501c P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:53.296521 7617 raft_consensus.cc:1261] T b4298fc9f7694af19716ef4a908a501c P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:53.296795 12575 consensus_queue.cc:1035] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:53.297076 12574 consensus_queue.cc:1035] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:27:53.297459 7416 consensus_peers.cc:485] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b4298fc9f7694af19716ef4a908a501c. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:53.300782 12574 raft_consensus.cc:2932] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:53.301443 7349 raft_consensus.cc:2932] T b4298fc9f7694af19716ef4a908a501c P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:53.304623 7617 raft_consensus.cc:2932] T b4298fc9f7694af19716ef4a908a501c P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:53.307169 4292 catalog_manager.cc:5484] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 reported cstate change: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:53.339851 7482 raft_consensus.cc:1203] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:53.432741 7349 consensus_queue.cc:237] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
W20240502 21:27:53.435611 7285 consensus_peers.cc:485] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6459a8c391249529d2672ab6c62a8af. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:53.435554 4440 raft_consensus.cc:1261] T c6459a8c391249529d2672ab6c62a8af P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:53.435808 7617 raft_consensus.cc:1261] T c6459a8c391249529d2672ab6c62a8af P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:53.436749 12489 consensus_queue.cc:1035] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:53.437049 12167 consensus_queue.cc:1035] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:53.440737 12167 raft_consensus.cc:2932] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:53.441350 4440 raft_consensus.cc:2932] T c6459a8c391249529d2672ab6c62a8af P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:53.445147 7617 raft_consensus.cc:2932] T c6459a8c391249529d2672ab6c62a8af P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:53.447621 4292 catalog_manager.cc:5484] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c reported cstate change: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:53.464241 4440 raft_consensus.cc:1203] T 84c8bdb300064adc86a4b23e07d512d3 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:53.488209 4440 consensus_queue.cc:237] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:53.489594 12672 ts_tablet_manager.cc:902] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04: Initiating tablet copy from peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
W20240502 21:27:53.497001 4376 consensus_peers.cc:485] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 -> Peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): Couldn't send request to peer 0ddd486109f5434cace13d60d633ac04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0914e3ce5a2d43788ba9d369227e4f66. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:53.497007 4838 raft_consensus.cc:1261] T 0914e3ce5a2d43788ba9d369227e4f66 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:53.497720 12464 consensus_queue.cc:1035] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:53.503561 7215 raft_consensus.cc:1261] T 0914e3ce5a2d43788ba9d369227e4f66 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:53.505051 12075 consensus_queue.cc:1035] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:53.508330 12672 tablet_copy_client.cc:308] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.65:42781
I20240502 21:27:53.508678 12464 raft_consensus.cc:2932] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:53.509310 7215 raft_consensus.cc:2932] T 0914e3ce5a2d43788ba9d369227e4f66 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:53.512513 4463 tablet_copy_service.cc:140] P 71e1375b799b439fa3c83ca763c039c1: Received BeginTabletCopySession request for tablet ca332df7aaef4ddd9fd3266e8003b55f from peer 0ddd486109f5434cace13d60d633ac04 ({username='slave'} at 127.29.82.72:46359)
I20240502 21:27:53.512661 4463 tablet_copy_service.cc:161] P 71e1375b799b439fa3c83ca763c039c1: Beginning new tablet copy session on tablet ca332df7aaef4ddd9fd3266e8003b55f from peer 0ddd486109f5434cace13d60d633ac04 at {username='slave'} at 127.29.82.72:46359: session id = 0ddd486109f5434cace13d60d633ac04-ca332df7aaef4ddd9fd3266e8003b55f
I20240502 21:27:53.515350 4463 tablet_copy_source_session.cc:205] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:53.517079 12672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca332df7aaef4ddd9fd3266e8003b55f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:53.520218 4838 raft_consensus.cc:2932] T 0914e3ce5a2d43788ba9d369227e4f66 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:53.522617 4292 catalog_manager.cc:5484] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: config changed from index -1 to 2, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:27:53.524698 12672 tablet_copy_client.cc:783] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:53.524906 12672 tablet_copy_client.cc:647] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:53.537853 12672 tablet_copy_client.cc:523] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:53.540395 12672 tablet_bootstrap.cc:492] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:53.552773 12672 tablet_bootstrap.cc:492] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:53.553135 12672 tablet_bootstrap.cc:492] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04: Bootstrap complete.
I20240502 21:27:53.553382 12672 ts_tablet_manager.cc:1367] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20240502 21:27:53.554059 12672 raft_consensus.cc:348] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:53.554286 12672 raft_consensus.cc:729] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:53.554482 12672 consensus_queue.cc:260] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:53.554563 7349 raft_consensus.cc:1203] T dd1cf8e447e54d6e907ffa0e4681515f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:53.555142 12672 ts_tablet_manager.cc:1398] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 21:27:53.556052 4463 tablet_copy_service.cc:342] P 71e1375b799b439fa3c83ca763c039c1: Request end of tablet copy session 0ddd486109f5434cace13d60d633ac04-ca332df7aaef4ddd9fd3266e8003b55f received from {username='slave'} at 127.29.82.72:46359
I20240502 21:27:53.556180 4463 tablet_copy_service.cc:433] P 71e1375b799b439fa3c83ca763c039c1: ending tablet copy session 0ddd486109f5434cace13d60d633ac04-ca332df7aaef4ddd9fd3266e8003b55f on tablet ca332df7aaef4ddd9fd3266e8003b55f with peer 0ddd486109f5434cace13d60d633ac04
I20240502 21:27:53.602279 7215 consensus_queue.cc:237] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:53.605029 7349 raft_consensus.cc:1261] T 1d3be8c5eaac423db11b282739cd1d40 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:27:53.605103 7151 consensus_peers.cc:485] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d3be8c5eaac423db11b282739cd1d40. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:53.605229 7482 raft_consensus.cc:1261] T 1d3be8c5eaac423db11b282739cd1d40 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:53.605901 12310 consensus_queue.cc:1035] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:53.606446 12310 consensus_queue.cc:1035] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:53.609613 12310 raft_consensus.cc:2932] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:53.610180 7349 raft_consensus.cc:2932] T 1d3be8c5eaac423db11b282739cd1d40 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:53.619141 4295 catalog_manager.cc:5484] T 1d3be8c5eaac423db11b282739cd1d40 P d0456c5e55a542128688904e003d403c reported cstate change: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:53.619472 7482 raft_consensus.cc:2932] T 1d3be8c5eaac423db11b282739cd1d40 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:53.699235 12725 ts_tablet_manager.cc:902] T 8991236419e642bfa438b9e45a1b8235 P 398c3c8f3195492c900439ad5e1931bb: Initiating tablet copy from peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:27:53.707228 12725 tablet_copy_client.cc:308] T 8991236419e642bfa438b9e45a1b8235 P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.72:41223
I20240502 21:27:53.707850 7638 tablet_copy_service.cc:140] P 0ddd486109f5434cace13d60d633ac04: Received BeginTabletCopySession request for tablet 8991236419e642bfa438b9e45a1b8235 from peer 398c3c8f3195492c900439ad5e1931bb ({username='slave'} at 127.29.82.67:57193)
I20240502 21:27:53.707985 7638 tablet_copy_service.cc:161] P 0ddd486109f5434cace13d60d633ac04: Beginning new tablet copy session on tablet 8991236419e642bfa438b9e45a1b8235 from peer 398c3c8f3195492c900439ad5e1931bb at {username='slave'} at 127.29.82.67:57193: session id = 398c3c8f3195492c900439ad5e1931bb-8991236419e642bfa438b9e45a1b8235
I20240502 21:27:53.710651 7638 tablet_copy_source_session.cc:205] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:53.712098 12725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8991236419e642bfa438b9e45a1b8235. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:53.718210 12725 tablet_copy_client.cc:783] T 8991236419e642bfa438b9e45a1b8235 P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:53.718865 12725 tablet_copy_client.cc:647] T 8991236419e642bfa438b9e45a1b8235 P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:53.721917 12725 tablet_copy_client.cc:523] T 8991236419e642bfa438b9e45a1b8235 P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:53.723675 4572 raft_consensus.cc:1203] T 29235795acc147f68d2ccce93564ebd3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:53.753731 12725 tablet_bootstrap.cc:492] T 8991236419e642bfa438b9e45a1b8235 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:53.773576 12725 tablet_bootstrap.cc:492] T 8991236419e642bfa438b9e45a1b8235 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:53.774606 12725 tablet_bootstrap.cc:492] T 8991236419e642bfa438b9e45a1b8235 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap complete.
I20240502 21:27:53.774895 12725 ts_tablet_manager.cc:1367] T 8991236419e642bfa438b9e45a1b8235 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.004s
I20240502 21:27:53.776340 12725 raft_consensus.cc:348] T 8991236419e642bfa438b9e45a1b8235 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:53.776664 12725 raft_consensus.cc:729] T 8991236419e642bfa438b9e45a1b8235 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:53.776955 12725 consensus_queue.cc:260] T 8991236419e642bfa438b9e45a1b8235 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:53.777763 12725 ts_tablet_manager.cc:1398] T 8991236419e642bfa438b9e45a1b8235 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:27:53.778589 7638 tablet_copy_service.cc:342] P 0ddd486109f5434cace13d60d633ac04: Request end of tablet copy session 398c3c8f3195492c900439ad5e1931bb-8991236419e642bfa438b9e45a1b8235 received from {username='slave'} at 127.29.82.67:57193
I20240502 21:27:53.778699 7638 tablet_copy_service.cc:433] P 0ddd486109f5434cace13d60d633ac04: ending tablet copy session 398c3c8f3195492c900439ad5e1931bb-8991236419e642bfa438b9e45a1b8235 on tablet 8991236419e642bfa438b9e45a1b8235 with peer 398c3c8f3195492c900439ad5e1931bb
I20240502 21:27:53.816205 12752 ts_tablet_manager.cc:902] T c6459a8c391249529d2672ab6c62a8af P 7a736a8ebb164ce0a051c0fccd76c3b8: Initiating tablet copy from peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:27:53.820441 12754 ts_tablet_manager.cc:902] T b4298fc9f7694af19716ef4a908a501c P 73a1f2ca1f544de9b0e3b37015f85328: Initiating tablet copy from peer 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:27:53.845435 12752 tablet_copy_client.cc:308] T c6459a8c391249529d2672ab6c62a8af P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.70:40719
I20240502 21:27:53.846400 7370 tablet_copy_service.cc:140] P d0456c5e55a542128688904e003d403c: Received BeginTabletCopySession request for tablet c6459a8c391249529d2672ab6c62a8af from peer 7a736a8ebb164ce0a051c0fccd76c3b8 ({username='slave'} at 127.29.82.66:56529)
I20240502 21:27:53.846567 7370 tablet_copy_service.cc:161] P d0456c5e55a542128688904e003d403c: Beginning new tablet copy session on tablet c6459a8c391249529d2672ab6c62a8af from peer 7a736a8ebb164ce0a051c0fccd76c3b8 at {username='slave'} at 127.29.82.66:56529: session id = 7a736a8ebb164ce0a051c0fccd76c3b8-c6459a8c391249529d2672ab6c62a8af
I20240502 21:27:53.849397 7370 tablet_copy_source_session.cc:205] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:53.853128 12752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6459a8c391249529d2672ab6c62a8af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:53.868748 12752 tablet_copy_client.cc:783] T c6459a8c391249529d2672ab6c62a8af P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:53.869192 12752 tablet_copy_client.cc:647] T c6459a8c391249529d2672ab6c62a8af P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:53.871311 12752 tablet_copy_client.cc:523] T c6459a8c391249529d2672ab6c62a8af P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:53.885573 12752 tablet_bootstrap.cc:492] T c6459a8c391249529d2672ab6c62a8af P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:27:53.915372 12752 tablet_bootstrap.cc:492] T c6459a8c391249529d2672ab6c62a8af P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:53.917387 12752 tablet_bootstrap.cc:492] T c6459a8c391249529d2672ab6c62a8af P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap complete.
I20240502 21:27:53.918948 12752 ts_tablet_manager.cc:1367] T c6459a8c391249529d2672ab6c62a8af P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.007s
I20240502 21:27:53.920018 12752 raft_consensus.cc:348] T c6459a8c391249529d2672ab6c62a8af P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:53.921250 12752 raft_consensus.cc:729] T c6459a8c391249529d2672ab6c62a8af P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:27:53.921728 12752 consensus_queue.cc:260] T c6459a8c391249529d2672ab6c62a8af P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:53.922901 12752 ts_tablet_manager.cc:1398] T c6459a8c391249529d2672ab6c62a8af P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 21:27:53.924310 7370 tablet_copy_service.cc:342] P d0456c5e55a542128688904e003d403c: Request end of tablet copy session 7a736a8ebb164ce0a051c0fccd76c3b8-c6459a8c391249529d2672ab6c62a8af received from {username='slave'} at 127.29.82.66:56529
I20240502 21:27:53.924441 7370 tablet_copy_service.cc:433] P d0456c5e55a542128688904e003d403c: ending tablet copy session 7a736a8ebb164ce0a051c0fccd76c3b8-c6459a8c391249529d2672ab6c62a8af on tablet c6459a8c391249529d2672ab6c62a8af with peer 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:27:53.924417 12754 tablet_copy_client.cc:308] T b4298fc9f7694af19716ef4a908a501c P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.71:33913
I20240502 21:27:53.927629 7505 tablet_copy_service.cc:140] P 31a72286397c4edc9a63eca3412be168: Received BeginTabletCopySession request for tablet b4298fc9f7694af19716ef4a908a501c from peer 73a1f2ca1f544de9b0e3b37015f85328 ({username='slave'} at 127.29.82.68:47963)
I20240502 21:27:53.927824 7505 tablet_copy_service.cc:161] P 31a72286397c4edc9a63eca3412be168: Beginning new tablet copy session on tablet b4298fc9f7694af19716ef4a908a501c from peer 73a1f2ca1f544de9b0e3b37015f85328 at {username='slave'} at 127.29.82.68:47963: session id = 73a1f2ca1f544de9b0e3b37015f85328-b4298fc9f7694af19716ef4a908a501c
I20240502 21:27:53.940595 7505 tablet_copy_source_session.cc:205] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:53.941953 12754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4298fc9f7694af19716ef4a908a501c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:53.952070 12754 tablet_copy_client.cc:783] T b4298fc9f7694af19716ef4a908a501c P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:53.952311 12754 tablet_copy_client.cc:647] T b4298fc9f7694af19716ef4a908a501c P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:53.971518 12754 tablet_copy_client.cc:523] T b4298fc9f7694af19716ef4a908a501c P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:53.976853 12754 tablet_bootstrap.cc:492] T b4298fc9f7694af19716ef4a908a501c P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:27:53.988826 12672 ts_tablet_manager.cc:902] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04: Initiating tablet copy from peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:27:53.991473 12754 tablet_bootstrap.cc:492] T b4298fc9f7694af19716ef4a908a501c P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:53.992008 12754 tablet_bootstrap.cc:492] T b4298fc9f7694af19716ef4a908a501c P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap complete.
I20240502 21:27:53.992331 12754 ts_tablet_manager.cc:1367] T b4298fc9f7694af19716ef4a908a501c P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.000s
I20240502 21:27:53.993278 12754 raft_consensus.cc:348] T b4298fc9f7694af19716ef4a908a501c P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:53.993598 12754 raft_consensus.cc:729] T b4298fc9f7694af19716ef4a908a501c P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:27:53.993886 12754 consensus_queue.cc:260] T b4298fc9f7694af19716ef4a908a501c P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:53.996793 12754 ts_tablet_manager.cc:1398] T b4298fc9f7694af19716ef4a908a501c P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:27:53.997843 7505 tablet_copy_service.cc:342] P 31a72286397c4edc9a63eca3412be168: Request end of tablet copy session 73a1f2ca1f544de9b0e3b37015f85328-b4298fc9f7694af19716ef4a908a501c received from {username='slave'} at 127.29.82.68:47963
I20240502 21:27:53.997958 7505 tablet_copy_service.cc:433] P 31a72286397c4edc9a63eca3412be168: ending tablet copy session 73a1f2ca1f544de9b0e3b37015f85328-b4298fc9f7694af19716ef4a908a501c on tablet b4298fc9f7694af19716ef4a908a501c with peer 73a1f2ca1f544de9b0e3b37015f85328
I20240502 21:27:54.011731 12672 tablet_copy_client.cc:308] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.65:42781
I20240502 21:27:54.012923 4463 tablet_copy_service.cc:140] P 71e1375b799b439fa3c83ca763c039c1: Received BeginTabletCopySession request for tablet 0914e3ce5a2d43788ba9d369227e4f66 from peer 0ddd486109f5434cace13d60d633ac04 ({username='slave'} at 127.29.82.72:46359)
I20240502 21:27:54.013093 4463 tablet_copy_service.cc:161] P 71e1375b799b439fa3c83ca763c039c1: Beginning new tablet copy session on tablet 0914e3ce5a2d43788ba9d369227e4f66 from peer 0ddd486109f5434cace13d60d633ac04 at {username='slave'} at 127.29.82.72:46359: session id = 0ddd486109f5434cace13d60d633ac04-0914e3ce5a2d43788ba9d369227e4f66
I20240502 21:27:54.015131 7617 raft_consensus.cc:1203] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:54.015949 4463 tablet_copy_source_session.cc:205] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:54.017302 12672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0914e3ce5a2d43788ba9d369227e4f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:54.023686 12672 tablet_copy_client.cc:783] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:54.023912 12672 tablet_copy_client.cc:647] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:54.027824 12672 tablet_copy_client.cc:523] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:54.032653 12672 tablet_bootstrap.cc:492] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:27:54.043223 12672 tablet_bootstrap.cc:492] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:54.043742 12672 tablet_bootstrap.cc:492] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04: Bootstrap complete.
I20240502 21:27:54.044054 12672 ts_tablet_manager.cc:1367] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.001s
I20240502 21:27:54.045303 12672 raft_consensus.cc:348] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:54.045606 12672 raft_consensus.cc:729] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:27:54.046002 12725 ts_tablet_manager.cc:902] T 1d3be8c5eaac423db11b282739cd1d40 P 398c3c8f3195492c900439ad5e1931bb: Initiating tablet copy from peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:27:54.045881 12672 consensus_queue.cc:260] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:27:54.046545 12672 ts_tablet_manager.cc:1398] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:27:54.049257 4463 tablet_copy_service.cc:342] P 71e1375b799b439fa3c83ca763c039c1: Request end of tablet copy session 0ddd486109f5434cace13d60d633ac04-0914e3ce5a2d43788ba9d369227e4f66 received from {username='slave'} at 127.29.82.72:46359
I20240502 21:27:54.049400 4463 tablet_copy_service.cc:433] P 71e1375b799b439fa3c83ca763c039c1: ending tablet copy session 0ddd486109f5434cace13d60d633ac04-0914e3ce5a2d43788ba9d369227e4f66 on tablet 0914e3ce5a2d43788ba9d369227e4f66 with peer 0ddd486109f5434cace13d60d633ac04
I20240502 21:27:54.100435 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:54.101596 12725 tablet_copy_client.cc:308] T 1d3be8c5eaac423db11b282739cd1d40 P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.69:33193
I20240502 21:27:54.102295 7236 tablet_copy_service.cc:140] P 39981a8dfe94462b9a73ba489af4c714: Received BeginTabletCopySession request for tablet 1d3be8c5eaac423db11b282739cd1d40 from peer 398c3c8f3195492c900439ad5e1931bb ({username='slave'} at 127.29.82.67:59759)
I20240502 21:27:54.102533 7236 tablet_copy_service.cc:161] P 39981a8dfe94462b9a73ba489af4c714: Beginning new tablet copy session on tablet 1d3be8c5eaac423db11b282739cd1d40 from peer 398c3c8f3195492c900439ad5e1931bb at {username='slave'} at 127.29.82.67:59759: session id = 398c3c8f3195492c900439ad5e1931bb-1d3be8c5eaac423db11b282739cd1d40
I20240502 21:27:54.112944 7236 tablet_copy_source_session.cc:205] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:54.116514 12725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3be8c5eaac423db11b282739cd1d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:54.126293 12725 tablet_copy_client.cc:783] T 1d3be8c5eaac423db11b282739cd1d40 P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:54.126632 12725 tablet_copy_client.cc:647] T 1d3be8c5eaac423db11b282739cd1d40 P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:54.129402 12725 tablet_copy_client.cc:523] T 1d3be8c5eaac423db11b282739cd1d40 P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:54.133947 12725 tablet_bootstrap.cc:492] T 1d3be8c5eaac423db11b282739cd1d40 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:27:54.159035 12725 tablet_bootstrap.cc:492] T 1d3be8c5eaac423db11b282739cd1d40 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:54.166340 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:54.171620 12725 tablet_bootstrap.cc:492] T 1d3be8c5eaac423db11b282739cd1d40 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap complete.
I20240502 21:27:54.171933 12725 ts_tablet_manager.cc:1367] T 1d3be8c5eaac423db11b282739cd1d40 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.038s user 0.008s sys 0.005s
I20240502 21:27:54.175194 12725 raft_consensus.cc:348] T 1d3be8c5eaac423db11b282739cd1d40 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:54.175385 12725 raft_consensus.cc:729] T 1d3be8c5eaac423db11b282739cd1d40 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:27:54.175628 12725 consensus_queue.cc:260] T 1d3be8c5eaac423db11b282739cd1d40 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:54.176172 12725 ts_tablet_manager.cc:1398] T 1d3be8c5eaac423db11b282739cd1d40 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:54.179272 7236 tablet_copy_service.cc:342] P 39981a8dfe94462b9a73ba489af4c714: Request end of tablet copy session 398c3c8f3195492c900439ad5e1931bb-1d3be8c5eaac423db11b282739cd1d40 received from {username='slave'} at 127.29.82.67:59759
I20240502 21:27:54.179419 7236 tablet_copy_service.cc:433] P 39981a8dfe94462b9a73ba489af4c714: ending tablet copy session 398c3c8f3195492c900439ad5e1931bb-1d3be8c5eaac423db11b282739cd1d40 on tablet 1d3be8c5eaac423db11b282739cd1d40 with peer 398c3c8f3195492c900439ad5e1931bb
I20240502 21:27:54.197052 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:54.232581 4838 raft_consensus.cc:1203] T b4298fc9f7694af19716ef4a908a501c P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:54.245074 4705 raft_consensus.cc:1203] T 8991236419e642bfa438b9e45a1b8235 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:54.247720 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:54.285897 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:54.285964 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:54.307495 7328 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240502 21:27:54.310174 4772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.70:40719, user_credentials={real_user=slave}} blocked reactor thread for 56434us
I20240502 21:27:54.327468 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:27:54.417459 7616 raft_consensus.cc:1203] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:54.437420 4574 raft_consensus.cc:1203] T c6459a8c391249529d2672ab6c62a8af P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:54.605293 4704 raft_consensus.cc:1203] T 1d3be8c5eaac423db11b282739cd1d40 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:54.766583 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:54.802685 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:54.866238 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:54.908627 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:54.918187 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:27:54.941787 7328 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 21:27:54.983474 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:55.027410 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:55.160326 7348 consensus_queue.cc:237] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:55.162220 4572 raft_consensus.cc:1261] T c6459a8c391249529d2672ab6c62a8af P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:55.162385 7616 raft_consensus.cc:1261] T c6459a8c391249529d2672ab6c62a8af P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:55.162856 12776 consensus_queue.cc:1035] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:55.163139 12652 consensus_queue.cc:1035] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:55.166054 12776 raft_consensus.cc:2932] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:55.166464 7616 raft_consensus.cc:2932] T c6459a8c391249529d2672ab6c62a8af P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:55.166800 4572 raft_consensus.cc:2932] T c6459a8c391249529d2672ab6c62a8af P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:55.173166 4294 catalog_manager.cc:5484] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c reported cstate change: config changed from index 2 to 3, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) evicted. New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:55.191493 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet c6459a8c391249529d2672ab6c62a8af with delete_type TABLET_DATA_TOMBSTONED (TS 71e1375b799b439fa3c83ca763c039c1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:35984
I20240502 21:27:55.204226 12916 tablet_replica.cc:330] T c6459a8c391249529d2672ab6c62a8af P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:27:55.204560 12916 raft_consensus.cc:2229] T c6459a8c391249529d2672ab6c62a8af P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:55.204857 12916 raft_consensus.cc:2258] T c6459a8c391249529d2672ab6c62a8af P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:55.205776 12916 ts_tablet_manager.cc:1875] T c6459a8c391249529d2672ab6c62a8af P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:55.213827 12916 ts_tablet_manager.cc:1888] T c6459a8c391249529d2672ab6c62a8af P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:27:55.213955 12916 log.cc:1196] T c6459a8c391249529d2672ab6c62a8af P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/c6459a8c391249529d2672ab6c62a8af
I20240502 21:27:55.214958 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet c6459a8c391249529d2672ab6c62a8af (table rebalancer_extra_table_24 [id=add5d1abdc46419a86ab49c59629bc2d]) successfully deleted
I20240502 21:27:55.589110 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:55.610177 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:55.651429 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:55.655560 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:55.677171 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:55.724226 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:55.761090 7328 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 21:27:55.761279 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:27:55.941653 7483 consensus_queue.cc:237] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:27:55.944231 4839 raft_consensus.cc:1261] T b4298fc9f7694af19716ef4a908a501c P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:55.944595 7617 raft_consensus.cc:1261] T b4298fc9f7694af19716ef4a908a501c P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:55.945127 12884 consensus_queue.cc:1035] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:55.945446 12574 consensus_queue.cc:1035] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:55.948246 12574 raft_consensus.cc:2932] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:55.948819 7617 raft_consensus.cc:2932] T b4298fc9f7694af19716ef4a908a501c P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:55.952641 4839 raft_consensus.cc:2932] T b4298fc9f7694af19716ef4a908a501c P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:55.955323 4295 catalog_manager.cc:5484] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 reported cstate change: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:27:55.971696 7328 tablet_service.cc:1514] Processing DeleteTablet for tablet b4298fc9f7694af19716ef4a908a501c with delete_type TABLET_DATA_TOMBSTONED (TS d0456c5e55a542128688904e003d403c not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47078
I20240502 21:27:55.972374 13015 tablet_replica.cc:330] T b4298fc9f7694af19716ef4a908a501c P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:27:55.972702 13015 raft_consensus.cc:2229] T b4298fc9f7694af19716ef4a908a501c P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:55.973177 13015 raft_consensus.cc:2258] T b4298fc9f7694af19716ef4a908a501c P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:55.974267 13015 ts_tablet_manager.cc:1875] T b4298fc9f7694af19716ef4a908a501c P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:55.981745 13015 ts_tablet_manager.cc:1888] T b4298fc9f7694af19716ef4a908a501c P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:27:55.981989 13015 log.cc:1196] T b4298fc9f7694af19716ef4a908a501c P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/b4298fc9f7694af19716ef4a908a501c
I20240502 21:27:55.983074 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet b4298fc9f7694af19716ef4a908a501c (table rebalancer_extra_table_20 [id=8a6e0f9c1e92442cbf43f55d3eef080f]) successfully deleted
I20240502 21:27:56.237238 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:56.270769 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:56.271157 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:56.364727 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:56.447510 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:56.460646 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:56.468585 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:27:56.475666 7328 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 21:27:56.632164 7617 consensus_queue.cc:237] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:27:56.634078 7350 raft_consensus.cc:1261] T 8991236419e642bfa438b9e45a1b8235 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:56.634368 4705 raft_consensus.cc:1261] T 8991236419e642bfa438b9e45a1b8235 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:56.634768 12981 consensus_queue.cc:1035] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:56.635061 12523 consensus_queue.cc:1035] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:56.638187 12523 raft_consensus.cc:2932] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:56.638564 7350 raft_consensus.cc:2932] T 8991236419e642bfa438b9e45a1b8235 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:56.642796 4705 raft_consensus.cc:2932] T 8991236419e642bfa438b9e45a1b8235 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:56.643738 4300 catalog_manager.cc:5484] T 8991236419e642bfa438b9e45a1b8235 P d0456c5e55a542128688904e003d403c reported cstate change: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:27:56.664260 7462 tablet_service.cc:1514] Processing DeleteTablet for tablet 8991236419e642bfa438b9e45a1b8235 with delete_type TABLET_DATA_TOMBSTONED (TS 31a72286397c4edc9a63eca3412be168 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:37182
I20240502 21:27:56.664955 13106 tablet_replica.cc:330] T 8991236419e642bfa438b9e45a1b8235 P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:27:56.665226 13106 raft_consensus.cc:2229] T 8991236419e642bfa438b9e45a1b8235 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:56.667412 13106 raft_consensus.cc:2258] T 8991236419e642bfa438b9e45a1b8235 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:56.668226 13106 ts_tablet_manager.cc:1875] T 8991236419e642bfa438b9e45a1b8235 P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:56.674767 13106 ts_tablet_manager.cc:1888] T 8991236419e642bfa438b9e45a1b8235 P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:27:56.674929 13106 log.cc:1196] T 8991236419e642bfa438b9e45a1b8235 P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/8991236419e642bfa438b9e45a1b8235
I20240502 21:27:56.675734 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet 8991236419e642bfa438b9e45a1b8235 (table rebalancer_extra_table_32 [id=9606abae682a4594ada645db2e56a7d4]) successfully deleted
I20240502 21:27:57.096482 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:57.149075 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:57.155370 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:57.161868 7328 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 21:27:57.175787 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:57.176960 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:57.210189 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:57.212424 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:27:57.444078 4838 consensus_queue.cc:237] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:57.445919 4572 raft_consensus.cc:1261] T dd1cf8e447e54d6e907ffa0e4681515f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:57.445955 7348 raft_consensus.cc:1261] T dd1cf8e447e54d6e907ffa0e4681515f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:57.446409 13127 consensus_queue.cc:1035] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:57.447013 12813 consensus_queue.cc:1035] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:57.449024 13127 raft_consensus.cc:2932] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:57.449774 4572 raft_consensus.cc:2932] T dd1cf8e447e54d6e907ffa0e4681515f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:57.450110 7348 raft_consensus.cc:2932] T dd1cf8e447e54d6e907ffa0e4681515f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:57.456724 4294 catalog_manager.cc:5484] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:57.465355 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet dd1cf8e447e54d6e907ffa0e4681515f with delete_type TABLET_DATA_TOMBSTONED (TS 398c3c8f3195492c900439ad5e1931bb not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44764
I20240502 21:27:57.466078 13200 tablet_replica.cc:330] T dd1cf8e447e54d6e907ffa0e4681515f P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:27:57.466286 13200 raft_consensus.cc:2229] T dd1cf8e447e54d6e907ffa0e4681515f P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:57.466518 13200 raft_consensus.cc:2258] T dd1cf8e447e54d6e907ffa0e4681515f P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:57.467094 13200 ts_tablet_manager.cc:1875] T dd1cf8e447e54d6e907ffa0e4681515f P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:57.472638 13200 ts_tablet_manager.cc:1888] T dd1cf8e447e54d6e907ffa0e4681515f P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:27:57.472754 13200 log.cc:1196] T dd1cf8e447e54d6e907ffa0e4681515f P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/dd1cf8e447e54d6e907ffa0e4681515f
I20240502 21:27:57.473476 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet dd1cf8e447e54d6e907ffa0e4681515f (table rebalance_test_table_5 [id=0b924d712d1446f6b103821acdc47c7b]) successfully deleted
I20240502 21:27:57.739228 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:57.758642 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:57.881915 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:57.897152 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:57.914506 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:57.959302 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:57.961787 7328 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 21:27:58.003854 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:27:58.198405 4442 consensus_queue.cc:237] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:27:58.203174 4572 raft_consensus.cc:1261] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:58.203202 7483 raft_consensus.cc:1261] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:58.204139 12075 consensus_queue.cc:1035] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:58.205574 12075 consensus_queue.cc:1035] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:58.209165 13282 raft_consensus.cc:2932] T 2eb73e73b64b4b0f9e5116f258f2e147 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:27:58.209769 4572 raft_consensus.cc:2932] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:27:58.210428 7483 raft_consensus.cc:2932] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:27:58.217778 4291 catalog_manager.cc:5484] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168 reported cstate change: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:27:58.241389 7194 tablet_service.cc:1514] Processing DeleteTablet for tablet 2eb73e73b64b4b0f9e5116f258f2e147 with delete_type TABLET_DATA_TOMBSTONED (TS 39981a8dfe94462b9a73ba489af4c714 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44652
I20240502 21:27:58.253405 13288 tablet_replica.cc:330] T 2eb73e73b64b4b0f9e5116f258f2e147 P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:27:58.253626 13288 raft_consensus.cc:2229] T 2eb73e73b64b4b0f9e5116f258f2e147 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:58.255507 13288 raft_consensus.cc:2258] T 2eb73e73b64b4b0f9e5116f258f2e147 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:58.257624 13288 ts_tablet_manager.cc:1875] T 2eb73e73b64b4b0f9e5116f258f2e147 P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:58.267230 13288 ts_tablet_manager.cc:1888] T 2eb73e73b64b4b0f9e5116f258f2e147 P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:27:58.267370 13288 log.cc:1196] T 2eb73e73b64b4b0f9e5116f258f2e147 P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/2eb73e73b64b4b0f9e5116f258f2e147
I20240502 21:27:58.268642 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet 2eb73e73b64b4b0f9e5116f258f2e147 (table rebalancer_extra_table_22 [id=50e84343189b44c5bd6158fbd7ba5394]) successfully deleted
I20240502 21:27:58.707738 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:58.715391 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:58.716347 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:58.732326 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:58.750491 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:58.777916 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:58.817875 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:27:58.923638 7328 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 21:27:59.029997 4442 consensus_queue.cc:237] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:27:59.032720 7348 raft_consensus.cc:1261] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:59.033270 4839 raft_consensus.cc:1261] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:59.033883 13282 consensus_queue.cc:1035] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:59.034178 12075 consensus_queue.cc:1035] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:59.037637 13282 raft_consensus.cc:2932] T 292bfad92b14456da85bdc874fb7e06e P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:59.040846 4839 raft_consensus.cc:2932] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:59.048987 4299 catalog_manager.cc:5484] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:59.057813 7348 raft_consensus.cc:2932] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:27:59.076118 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet 292bfad92b14456da85bdc874fb7e06e with delete_type TABLET_DATA_TOMBSTONED (TS 398c3c8f3195492c900439ad5e1931bb not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44764
I20240502 21:27:59.088452 13385 tablet_replica.cc:330] T 292bfad92b14456da85bdc874fb7e06e P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:27:59.088835 13385 raft_consensus.cc:2229] T 292bfad92b14456da85bdc874fb7e06e P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:59.089208 13385 raft_consensus.cc:2258] T 292bfad92b14456da85bdc874fb7e06e P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:59.090394 13385 ts_tablet_manager.cc:1875] T 292bfad92b14456da85bdc874fb7e06e P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:59.097628 13385 ts_tablet_manager.cc:1888] T 292bfad92b14456da85bdc874fb7e06e P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:27:59.097826 13385 log.cc:1196] T 292bfad92b14456da85bdc874fb7e06e P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/292bfad92b14456da85bdc874fb7e06e
I20240502 21:27:59.100621 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet 292bfad92b14456da85bdc874fb7e06e (table rebalance_test_table_11 [id=6578c35615d249838e00b267cf4291d8]) successfully deleted
I20240502 21:27:59.429956 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:59.434303 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:59.441260 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:27:59.445032 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:59.463353 7328 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 21:27:59.474354 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:27:59.492043 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:27:59.510439 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:27:59.673833 4838 consensus_queue.cc:237] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:27:59.675920 4575 raft_consensus.cc:1261] T a417562f482e4d468095fbe833056233 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:59.676450 7212 raft_consensus.cc:1261] T a417562f482e4d468095fbe833056233 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:59.676688 12813 consensus_queue.cc:1035] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:59.677276 12813 consensus_queue.cc:1035] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:59.679432 13422 raft_consensus.cc:2932] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:59.680117 4575 raft_consensus.cc:2932] T a417562f482e4d468095fbe833056233 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:59.687295 4299 catalog_manager.cc:5484] T a417562f482e4d468095fbe833056233 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:59.693192 7215 raft_consensus.cc:2932] T a417562f482e4d468095fbe833056233 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:27:59.701261 7597 tablet_service.cc:1514] Processing DeleteTablet for tablet a417562f482e4d468095fbe833056233 with delete_type TABLET_DATA_TOMBSTONED (TS 0ddd486109f5434cace13d60d633ac04 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44444
I20240502 21:27:59.702042 13465 tablet_replica.cc:330] T a417562f482e4d468095fbe833056233 P 0ddd486109f5434cace13d60d633ac04: stopping tablet replica
I20240502 21:27:59.702739 13465 raft_consensus.cc:2229] T a417562f482e4d468095fbe833056233 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:59.703089 13465 raft_consensus.cc:2258] T a417562f482e4d468095fbe833056233 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:59.704406 13465 ts_tablet_manager.cc:1875] T a417562f482e4d468095fbe833056233 P 0ddd486109f5434cace13d60d633ac04: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:59.708873 13465 ts_tablet_manager.cc:1888] T a417562f482e4d468095fbe833056233 P 0ddd486109f5434cace13d60d633ac04: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:27:59.708987 13465 log.cc:1196] T a417562f482e4d468095fbe833056233 P 0ddd486109f5434cace13d60d633ac04: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/wals/a417562f482e4d468095fbe833056233
I20240502 21:27:59.709900 4285 catalog_manager.cc:4830] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): tablet a417562f482e4d468095fbe833056233 (table rebalancer_extra_table_14 [id=e7b935c5901749939ea064dc2f41c619]) successfully deleted
I20240502 21:28:00.085366 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:28:00.096993 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:00.119227 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:00.159407 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:28:00.179363 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:00.199924 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:28:00.227797 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:00.293053 7328 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 21:28:00.425026 4705 consensus_queue.cc:237] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:28:00.428764 7212 raft_consensus.cc:1261] T 3a1fadb11cd6436a94e4a67a3aad17ba P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:00.428738 4574 raft_consensus.cc:1261] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:00.429250 12351 consensus_queue.cc:1035] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:00.429524 13521 consensus_queue.cc:1035] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:00.432922 13521 raft_consensus.cc:2932] T 3a1fadb11cd6436a94e4a67a3aad17ba P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:00.433583 4573 raft_consensus.cc:2932] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:00.433928 7212 raft_consensus.cc:2932] T 3a1fadb11cd6436a94e4a67a3aad17ba P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:00.439882 4292 catalog_manager.cc:5484] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: config changed from index 4 to 5, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) evicted. New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:00.452997 4817 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a1fadb11cd6436a94e4a67a3aad17ba with delete_type TABLET_DATA_TOMBSTONED (TS 73a1f2ca1f544de9b0e3b37015f85328 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56326
I20240502 21:28:00.458359 13551 tablet_replica.cc:330] T 3a1fadb11cd6436a94e4a67a3aad17ba P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:28:00.459239 13551 raft_consensus.cc:2229] T 3a1fadb11cd6436a94e4a67a3aad17ba P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:00.459602 13551 raft_consensus.cc:2258] T 3a1fadb11cd6436a94e4a67a3aad17ba P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:00.461447 13551 ts_tablet_manager.cc:1875] T 3a1fadb11cd6436a94e4a67a3aad17ba P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:00.470356 13551 ts_tablet_manager.cc:1888] T 3a1fadb11cd6436a94e4a67a3aad17ba P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:28:00.470477 13551 log.cc:1196] T 3a1fadb11cd6436a94e4a67a3aad17ba P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/3a1fadb11cd6436a94e4a67a3aad17ba
I20240502 21:28:00.471237 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet 3a1fadb11cd6436a94e4a67a3aad17ba (table rebalance_test_table_19 [id=879d91ac491a4b2dbed04af44dc5e8de]) successfully deleted
I20240502 21:28:00.696959 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:00.698307 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:28:00.701246 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:00.733453 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:28:00.764250 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:00.851562 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:00.855981 7328 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240502 21:28:00.955469 7148 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.68:42449, user_credentials={real_user=slave}} blocked reactor thread for 72749.4us
I20240502 21:28:00.968991 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:28:01.125392 7616 consensus_queue.cc:237] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:28:01.127321 4442 raft_consensus.cc:1261] T 9e5434cb329e4bea85317d18f6e6867d P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:01.127337 7212 raft_consensus.cc:1261] T 9e5434cb329e4bea85317d18f6e6867d P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:01.127885 13099 consensus_queue.cc:1035] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:01.128187 13497 consensus_queue.cc:1035] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:01.131479 13497 raft_consensus.cc:2932] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:01.132093 7212 raft_consensus.cc:2932] T 9e5434cb329e4bea85317d18f6e6867d P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:01.137737 4294 catalog_manager.cc:5484] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 reported cstate change: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:01.145345 4442 raft_consensus.cc:2932] T 9e5434cb329e4bea85317d18f6e6867d P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:01.147759 7462 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e5434cb329e4bea85317d18f6e6867d with delete_type TABLET_DATA_TOMBSTONED (TS 31a72286397c4edc9a63eca3412be168 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:37182
I20240502 21:28:01.148550 13637 tablet_replica.cc:330] T 9e5434cb329e4bea85317d18f6e6867d P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:28:01.148800 13637 raft_consensus.cc:2229] T 9e5434cb329e4bea85317d18f6e6867d P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:01.149061 13637 raft_consensus.cc:2258] T 9e5434cb329e4bea85317d18f6e6867d P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:01.149786 13637 ts_tablet_manager.cc:1875] T 9e5434cb329e4bea85317d18f6e6867d P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:01.155752 13637 ts_tablet_manager.cc:1888] T 9e5434cb329e4bea85317d18f6e6867d P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:01.155895 13637 log.cc:1196] T 9e5434cb329e4bea85317d18f6e6867d P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/9e5434cb329e4bea85317d18f6e6867d
I20240502 21:28:01.156741 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet 9e5434cb329e4bea85317d18f6e6867d (table rebalancer_extra_table_28 [id=37d345f972f34b419de6c03d44f832d8]) successfully deleted
I20240502 21:28:01.568738 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:28:01.592017 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:01.630177 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:01.752460 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:28:01.763023 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:28:01.784143 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:01.789466 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:01.854017 7328 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 21:28:01.973883 4705 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b7c60d990751440392ab6e5feb0d8498"
dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60710
I20240502 21:28:01.974186 4705 raft_consensus.cc:595] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Received request to transfer leadership
I20240502 21:28:02.178031 13521 raft_consensus.cc:979] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb: : Instructing follower d0456c5e55a542128688904e003d403c to start an election
I20240502 21:28:02.178206 13521 raft_consensus.cc:1067] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Signalling peer d0456c5e55a542128688904e003d403c to start an election
I20240502 21:28:02.179469 7350 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b7c60d990751440392ab6e5feb0d8498"
dest_uuid: "d0456c5e55a542128688904e003d403c"
from {username='slave'} at 127.29.82.67:52175
I20240502 21:28:02.179756 7350 raft_consensus.cc:482] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:28:02.179867 7350 raft_consensus.cc:3037] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:02.182049 7350 raft_consensus.cc:504] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:28:02.185144 7350 leader_election.cc:288] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 election: Requested vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:28:02.185547 4704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c60d990751440392ab6e5feb0d8498" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:28:02.185722 4704 raft_consensus.cc:3032] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:28:02.185788 4704 raft_consensus.cc:729] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Running, Role: LEADER
I20240502 21:28:02.186055 4704 consensus_queue.cc:260] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:28:02.186478 4704 raft_consensus.cc:3037] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:02.188814 4704 raft_consensus.cc:2454] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 2.
I20240502 21:28:02.189708 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c60d990751440392ab6e5feb0d8498" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:28:02.189944 4440 raft_consensus.cc:3037] T b7c60d990751440392ab6e5feb0d8498 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:02.191510 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c60d990751440392ab6e5feb0d8498" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:28:02.191736 4838 raft_consensus.cc:3037] T b7c60d990751440392ab6e5feb0d8498 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:02.192762 4440 raft_consensus.cc:2454] T b7c60d990751440392ab6e5feb0d8498 P 71e1375b799b439fa3c83ca763c039c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 2.
I20240502 21:28:02.193286 7281 leader_election.cc:302] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:28:02.193729 13356 raft_consensus.cc:2781] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:28:02.193848 13356 raft_consensus.cc:686] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c [term 2 LEADER]: Becoming Leader. State: Replica: d0456c5e55a542128688904e003d403c, State: Running, Role: LEADER
I20240502 21:28:02.194132 13356 consensus_queue.cc:237] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:28:02.196650 4838 raft_consensus.cc:2454] T b7c60d990751440392ab6e5feb0d8498 P 73a1f2ca1f544de9b0e3b37015f85328 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 2.
I20240502 21:28:02.203727 4297 catalog_manager.cc:5484] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c reported cstate change: term changed from 1 to 2, leader changed from 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) to d0456c5e55a542128688904e003d403c (127.29.82.70). New cstate: current_term: 2 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:02.245883 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:28:02.267513 4686 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:02.279613 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:28:02.294829 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:02.322469 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:02.344837 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:28:02.463608 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:02.468506 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:02.612164 4704 raft_consensus.cc:1261] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [term 2 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:28:02.612830 12776 consensus_queue.cc:1035] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:02.639182 4838 raft_consensus.cc:1261] T b7c60d990751440392ab6e5feb0d8498 P 73a1f2ca1f544de9b0e3b37015f85328 [term 2 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:28:02.640494 12776 consensus_queue.cc:1035] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:02.643680 4442 raft_consensus.cc:1261] T b7c60d990751440392ab6e5feb0d8498 P 71e1375b799b439fa3c83ca763c039c1 [term 2 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:28:02.644361 12776 consensus_queue.cc:1035] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:02.955845 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:28:02.977447 4686 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:02.987753 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:03.052155 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:03.074616 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:28:03.090755 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:03.099026 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:03.110390 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:28:03.286330 7349 consensus_queue.cc:237] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:28:03.288661 4573 raft_consensus.cc:1261] T 180e178d0c714a35b8ca8843fad466c0 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:03.289198 7483 raft_consensus.cc:1261] T 180e178d0c714a35b8ca8843fad466c0 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:03.289767 13356 consensus_queue.cc:1035] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:03.290045 13809 consensus_queue.cc:1035] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:03.293569 13809 raft_consensus.cc:2932] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:03.294309 4573 raft_consensus.cc:2932] T 180e178d0c714a35b8ca8843fad466c0 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:03.296221 7483 raft_consensus.cc:2932] T 180e178d0c714a35b8ca8843fad466c0 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:03.301028 4299 catalog_manager.cc:5484] T 180e178d0c714a35b8ca8843fad466c0 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:03.312002 7597 tablet_service.cc:1514] Processing DeleteTablet for tablet 180e178d0c714a35b8ca8843fad466c0 with delete_type TABLET_DATA_TOMBSTONED (TS 0ddd486109f5434cace13d60d633ac04 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44444
I20240502 21:28:03.331740 13881 tablet_replica.cc:330] T 180e178d0c714a35b8ca8843fad466c0 P 0ddd486109f5434cace13d60d633ac04: stopping tablet replica
I20240502 21:28:03.332178 13881 raft_consensus.cc:2229] T 180e178d0c714a35b8ca8843fad466c0 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:03.332557 13881 raft_consensus.cc:2258] T 180e178d0c714a35b8ca8843fad466c0 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:03.333490 13881 ts_tablet_manager.cc:1875] T 180e178d0c714a35b8ca8843fad466c0 P 0ddd486109f5434cace13d60d633ac04: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:03.341104 13881 ts_tablet_manager.cc:1888] T 180e178d0c714a35b8ca8843fad466c0 P 0ddd486109f5434cace13d60d633ac04: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:03.341233 13881 log.cc:1196] T 180e178d0c714a35b8ca8843fad466c0 P 0ddd486109f5434cace13d60d633ac04: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/wals/180e178d0c714a35b8ca8843fad466c0
I20240502 21:28:03.342274 4285 catalog_manager.cc:4830] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): tablet 180e178d0c714a35b8ca8843fad466c0 (table rebalancer_extra_table_60 [id=c758184ce285455e9f74bd2b6fd9dfc7]) successfully deleted
I20240502 21:28:03.642829 4686 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:03.655844 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:03.667757 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 21:28:03.740371 7149 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.72:41223, user_credentials={real_user=slave}} blocked reactor thread for 46499.1us
I20240502 21:28:03.758458 7194 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 21:28:03.775835 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:03.832996 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:03.890846 4817 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 21:28:03.904089 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:04.032768 7215 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a5d47492f64f49d584246fc5007281aa"
dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48874
I20240502 21:28:04.033077 7215 raft_consensus.cc:595] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:28:04.075235 13938 raft_consensus.cc:979] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714: : Instructing follower 73a1f2ca1f544de9b0e3b37015f85328 to start an election
I20240502 21:28:04.075414 13938 raft_consensus.cc:1067] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Signalling peer 73a1f2ca1f544de9b0e3b37015f85328 to start an election
I20240502 21:28:04.076385 4839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a5d47492f64f49d584246fc5007281aa"
dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
from {username='slave'} at 127.29.82.69:57807
I20240502 21:28:04.076648 4839 raft_consensus.cc:482] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:28:04.076737 4839 raft_consensus.cc:3037] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:04.079438 4839 raft_consensus.cc:504] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:28:04.080605 4839 leader_election.cc:288] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 2 election: Requested vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:28:04.081398 7616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d47492f64f49d584246fc5007281aa" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:28:04.081442 7215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d47492f64f49d584246fc5007281aa" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:28:04.081631 7616 raft_consensus.cc:3037] T a5d47492f64f49d584246fc5007281aa P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:04.081630 7215 raft_consensus.cc:3032] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:28:04.081703 7215 raft_consensus.cc:729] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:28:04.082026 7215 consensus_queue.cc:260] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:28:04.082486 7215 raft_consensus.cc:3037] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:04.083817 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d47492f64f49d584246fc5007281aa" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:28:04.084015 7483 raft_consensus.cc:3037] T a5d47492f64f49d584246fc5007281aa P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:04.084407 7616 raft_consensus.cc:2454] T a5d47492f64f49d584246fc5007281aa P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 2.
I20240502 21:28:04.086427 7483 raft_consensus.cc:2454] T a5d47492f64f49d584246fc5007281aa P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 2.
I20240502 21:28:04.088721 4771 leader_election.cc:302] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:28:04.089190 13693 raft_consensus.cc:2781] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:28:04.089329 13693 raft_consensus.cc:686] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328 [term 2 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:28:04.089623 13693 consensus_queue.cc:237] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:28:04.097558 7215 raft_consensus.cc:2454] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 2.
I20240502 21:28:04.102273 4298 catalog_manager.cc:5484] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 1 to 2, leader changed from 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 2 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:04.341545 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:28:04.368148 7194 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 21:28:04.383682 4686 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:04.406424 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:04.413172 4817 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 21:28:04.439980 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:04.456580 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:04.541074 7482 raft_consensus.cc:1261] T a5d47492f64f49d584246fc5007281aa P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 21:28:04.541577 13693 consensus_queue.cc:1035] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:04.550338 7617 raft_consensus.cc:1261] T a5d47492f64f49d584246fc5007281aa P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 21:28:04.550913 13693 consensus_queue.cc:1035] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:04.554773 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:04.633234 7212 raft_consensus.cc:1261] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [term 2 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 21:28:04.635082 13422 consensus_queue.cc:1035] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 21:28:05.195191 7285 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.66:35781, user_credentials={real_user=slave}} blocked reactor thread for 93027.9us
I20240502 21:28:05.237306 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:28:05.262991 4686 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 21:28:05.293553 4771 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.72:41223, user_credentials={real_user=slave}} blocked reactor thread for 109400us
I20240502 21:28:05.300824 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:05.334225 7194 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 21:28:05.335994 4773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.66:35781, user_credentials={real_user=slave}} blocked reactor thread for 152289us
I20240502 21:28:05.349996 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:05.394984 4817 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 21:28:05.440606 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:05.483114 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:05.618853 4442 consensus_queue.cc:237] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:28:05.621027 4839 raft_consensus.cc:1261] T 0914e3ce5a2d43788ba9d369227e4f66 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:05.621048 7616 raft_consensus.cc:1261] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:05.621613 13944 consensus_queue.cc:1035] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:05.621928 13611 consensus_queue.cc:1035] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:05.626207 13944 raft_consensus.cc:2932] T 0914e3ce5a2d43788ba9d369227e4f66 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:05.627276 4838 raft_consensus.cc:2932] T 0914e3ce5a2d43788ba9d369227e4f66 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:05.629217 7616 raft_consensus.cc:2932] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:05.637265 4300 catalog_manager.cc:5484] T 0914e3ce5a2d43788ba9d369227e4f66 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:05.649327 7194 tablet_service.cc:1514] Processing DeleteTablet for tablet 0914e3ce5a2d43788ba9d369227e4f66 with delete_type TABLET_DATA_TOMBSTONED (TS 39981a8dfe94462b9a73ba489af4c714 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44652
I20240502 21:28:05.650141 14141 tablet_replica.cc:330] T 0914e3ce5a2d43788ba9d369227e4f66 P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:28:05.650682 14141 raft_consensus.cc:2229] T 0914e3ce5a2d43788ba9d369227e4f66 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:05.651019 14141 raft_consensus.cc:2258] T 0914e3ce5a2d43788ba9d369227e4f66 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:05.652063 14141 ts_tablet_manager.cc:1875] T 0914e3ce5a2d43788ba9d369227e4f66 P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:05.660796 14141 ts_tablet_manager.cc:1888] T 0914e3ce5a2d43788ba9d369227e4f66 P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:05.661026 14141 log.cc:1196] T 0914e3ce5a2d43788ba9d369227e4f66 P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/0914e3ce5a2d43788ba9d369227e4f66
I20240502 21:28:05.662072 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet 0914e3ce5a2d43788ba9d369227e4f66 (table rebalancer_extra_table_62 [id=2f504526d1174ad1ad3808fc9c728ce7]) successfully deleted
I20240502 21:28:06.071359 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:06.075268 4686 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:06.092736 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:06.106541 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:06.118341 7194 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 21:28:06.133221 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:06.139518 4817 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 21:28:06.166479 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:28:06.398759 7350 consensus_queue.cc:237] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:28:06.411020 4442 raft_consensus.cc:1261] T 84c8bdb300064adc86a4b23e07d512d3 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:06.411291 4575 raft_consensus.cc:1261] T 84c8bdb300064adc86a4b23e07d512d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:06.412113 14100 consensus_queue.cc:1035] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:06.412693 14100 consensus_queue.cc:1035] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:06.418869 14212 raft_consensus.cc:2932] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:06.419437 4442 raft_consensus.cc:2932] T 84c8bdb300064adc86a4b23e07d512d3 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:06.421396 4575 raft_consensus.cc:2932] T 84c8bdb300064adc86a4b23e07d512d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:06.426725 4292 catalog_manager.cc:5484] T 84c8bdb300064adc86a4b23e07d512d3 P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: config changed from index 2 to 3, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) evicted. New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:06.443730 4817 tablet_service.cc:1514] Processing DeleteTablet for tablet 84c8bdb300064adc86a4b23e07d512d3 with delete_type TABLET_DATA_TOMBSTONED (TS 73a1f2ca1f544de9b0e3b37015f85328 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:56326
I20240502 21:28:06.460173 14241 tablet_replica.cc:330] T 84c8bdb300064adc86a4b23e07d512d3 P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:28:06.460438 14241 raft_consensus.cc:2229] T 84c8bdb300064adc86a4b23e07d512d3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:06.460656 14241 raft_consensus.cc:2258] T 84c8bdb300064adc86a4b23e07d512d3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:06.461334 14241 ts_tablet_manager.cc:1875] T 84c8bdb300064adc86a4b23e07d512d3 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:06.471940 14241 ts_tablet_manager.cc:1888] T 84c8bdb300064adc86a4b23e07d512d3 P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:06.472100 14241 log.cc:1196] T 84c8bdb300064adc86a4b23e07d512d3 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/84c8bdb300064adc86a4b23e07d512d3
I20240502 21:28:06.472950 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet 84c8bdb300064adc86a4b23e07d512d3 (table rebalancer_extra_table_40 [id=13f15c7aba6346b19da83861cfdb5f07]) successfully deleted
I20240502 21:28:06.736182 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:28:06.794103 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:06.834244 7194 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 21:28:06.879935 4817 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 21:28:06.914984 4376 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.72:41223, user_credentials={real_user=slave}} blocked reactor thread for 47471.5us
I20240502 21:28:06.929919 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:06.929960 4686 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:06.964561 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:07.010757 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:07.165709 7616 consensus_queue.cc:237] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:28:07.167737 4704 raft_consensus.cc:1261] T d09a80f79473469e92310ded0f4edfdd P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:07.167872 4442 raft_consensus.cc:1261] T d09a80f79473469e92310ded0f4edfdd P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:07.168345 13497 consensus_queue.cc:1035] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:07.168979 13497 consensus_queue.cc:1035] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:07.170874 14315 raft_consensus.cc:2932] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:07.171669 4705 raft_consensus.cc:2932] T d09a80f79473469e92310ded0f4edfdd P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:07.176818 4442 raft_consensus.cc:2932] T d09a80f79473469e92310ded0f4edfdd P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:07.179128 4298 catalog_manager.cc:5484] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 reported cstate change: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:07.221485 7462 tablet_service.cc:1514] Processing DeleteTablet for tablet d09a80f79473469e92310ded0f4edfdd with delete_type TABLET_DATA_TOMBSTONED (TS 31a72286397c4edc9a63eca3412be168 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:37182
I20240502 21:28:07.224498 14350 tablet_replica.cc:330] T d09a80f79473469e92310ded0f4edfdd P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:28:07.224912 14350 raft_consensus.cc:2229] T d09a80f79473469e92310ded0f4edfdd P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:07.225158 14350 raft_consensus.cc:2258] T d09a80f79473469e92310ded0f4edfdd P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:07.226227 14350 ts_tablet_manager.cc:1875] T d09a80f79473469e92310ded0f4edfdd P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:07.231842 14350 ts_tablet_manager.cc:1888] T d09a80f79473469e92310ded0f4edfdd P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:07.232021 14350 log.cc:1196] T d09a80f79473469e92310ded0f4edfdd P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/d09a80f79473469e92310ded0f4edfdd
I20240502 21:28:07.232714 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet d09a80f79473469e92310ded0f4edfdd (table rebalancer_extra_table_18 [id=1488a4e68a204d09a88b4a05c1c2cc24]) successfully deleted
I20240502 21:28:07.565317 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:28:07.593132 4686 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:07.599148 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:07.626241 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:07.661001 7194 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 21:28:07.712136 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:07.719714 4817 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 21:28:07.752213 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:07.898350 4440 consensus_queue.cc:237] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:28:07.900801 7616 raft_consensus.cc:1261] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:07.901129 7349 raft_consensus.cc:1261] T ca332df7aaef4ddd9fd3266e8003b55f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:07.901364 13944 consensus_queue.cc:1035] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:07.901630 14313 consensus_queue.cc:1035] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:07.904721 14313 raft_consensus.cc:2932] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:07.905555 7349 raft_consensus.cc:2932] T ca332df7aaef4ddd9fd3266e8003b55f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:07.906730 7616 raft_consensus.cc:2932] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:07.911916 4293 catalog_manager.cc:5484] T ca332df7aaef4ddd9fd3266e8003b55f P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:07.949172 7194 tablet_service.cc:1514] Processing DeleteTablet for tablet ca332df7aaef4ddd9fd3266e8003b55f with delete_type TABLET_DATA_TOMBSTONED (TS 39981a8dfe94462b9a73ba489af4c714 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44652
I20240502 21:28:07.952121 14451 tablet_replica.cc:330] T ca332df7aaef4ddd9fd3266e8003b55f P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:28:07.952328 14451 raft_consensus.cc:2229] T ca332df7aaef4ddd9fd3266e8003b55f P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:07.952509 14451 raft_consensus.cc:2258] T ca332df7aaef4ddd9fd3266e8003b55f P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:07.953032 14451 ts_tablet_manager.cc:1875] T ca332df7aaef4ddd9fd3266e8003b55f P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:07.958200 14451 ts_tablet_manager.cc:1888] T ca332df7aaef4ddd9fd3266e8003b55f P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:07.958354 14451 log.cc:1196] T ca332df7aaef4ddd9fd3266e8003b55f P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/ca332df7aaef4ddd9fd3266e8003b55f
I20240502 21:28:07.959075 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet ca332df7aaef4ddd9fd3266e8003b55f (table rebalancer_extra_table_22 [id=50e84343189b44c5bd6158fbd7ba5394]) successfully deleted
I20240502 21:28:08.220193 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:28:08.234057 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:08.244890 7194 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 21:28:08.274890 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:08.291173 4686 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:08.327105 4817 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 21:28:08.335657 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:08.411128 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240502 21:28:08.417802 7148 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.68:42449, user_credentials={real_user=slave}} blocked reactor thread for 124507us
I20240502 21:28:08.527273 7616 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "29235795acc147f68d2ccce93564ebd3"
dest_uuid: "0ddd486109f5434cace13d60d633ac04"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59050
I20240502 21:28:08.527519 7616 raft_consensus.cc:595] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:28:08.795455 7597 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:28:08.811125 4686 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:08.842330 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:08.848891 13497 raft_consensus.cc:979] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04: : Instructing follower 31a72286397c4edc9a63eca3412be168 to start an election
I20240502 21:28:08.849040 13497 raft_consensus.cc:1067] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Signalling peer 31a72286397c4edc9a63eca3412be168 to start an election
I20240502 21:28:08.850025 7483 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "29235795acc147f68d2ccce93564ebd3"
dest_uuid: "31a72286397c4edc9a63eca3412be168"
from {username='slave'} at 127.29.82.72:45539
I20240502 21:28:08.850253 7483 raft_consensus.cc:482] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:28:08.850339 7483 raft_consensus.cc:3037] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:08.853299 7483 raft_consensus.cc:504] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:28:08.854422 7483 leader_election.cc:288] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:28:08.856442 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29235795acc147f68d2ccce93564ebd3" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:28:08.856662 4705 raft_consensus.cc:3037] T 29235795acc147f68d2ccce93564ebd3 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:08.859333 4705 raft_consensus.cc:2454] T 29235795acc147f68d2ccce93564ebd3 P 398c3c8f3195492c900439ad5e1931bb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 2.
I20240502 21:28:08.859807 4703 raft_consensus.cc:1226] T 29235795acc147f68d2ccce93564ebd3 P 398c3c8f3195492c900439ad5e1931bb [term 2 FOLLOWER]: Rejecting Update request from peer 0ddd486109f5434cace13d60d633ac04 for earlier term 1. Current term is 2. Ops: []
I20240502 21:28:08.860468 14315 consensus_queue.cc:1046] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:08.860774 14315 raft_consensus.cc:3032] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:28:08.860862 14315 raft_consensus.cc:729] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Running, Role: LEADER
I20240502 21:28:08.861145 14315 consensus_queue.cc:260] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:28:08.861635 14315 raft_consensus.cc:3037] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:08.871274 7616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29235795acc147f68d2ccce93564ebd3" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:28:08.872573 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29235795acc147f68d2ccce93564ebd3" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:28:08.872761 4575 raft_consensus.cc:3037] T 29235795acc147f68d2ccce93564ebd3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:08.877671 7616 raft_consensus.cc:2454] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 2.
I20240502 21:28:08.881603 7420 leader_election.cc:302] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168, 398c3c8f3195492c900439ad5e1931bb; no voters:
I20240502 21:28:08.881870 14502 raft_consensus.cc:2781] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:28:08.881994 14502 raft_consensus.cc:686] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168 [term 2 LEADER]: Becoming Leader. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Running, Role: LEADER
I20240502 21:28:08.882316 14502 consensus_queue.cc:237] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:28:08.890003 4575 raft_consensus.cc:2454] T 29235795acc147f68d2ccce93564ebd3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 2.
I20240502 21:28:08.892412 4296 catalog_manager.cc:5484] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168 reported cstate change: term changed from 1 to 2, leader changed from 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) to 31a72286397c4edc9a63eca3412be168 (127.29.82.71). New cstate: current_term: 2 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:08.949945 7194 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 21:28:09.009900 4817 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 21:28:09.048207 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:09.057688 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:09.121277 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:09.242456 4440 consensus_queue.cc:237] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
W20240502 21:28:09.248437 4375 consensus_peers.cc:485] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1 -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb2cc722452f4d9798353d1bfae4bd4f. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:09.248533 7482 raft_consensus.cc:1261] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:09.249172 14313 consensus_queue.cc:1035] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:09.249300 7350 raft_consensus.cc:1261] T fb2cc722452f4d9798353d1bfae4bd4f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:09.249840 14313 consensus_queue.cc:1035] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:09.252359 14599 raft_consensus.cc:2932] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:09.252872 7350 raft_consensus.cc:2932] T fb2cc722452f4d9798353d1bfae4bd4f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:09.261162 4298 catalog_manager.cc:5484] T fb2cc722452f4d9798353d1bfae4bd4f P d0456c5e55a542128688904e003d403c reported cstate change: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:09.262697 4704 raft_consensus.cc:1261] T 29235795acc147f68d2ccce93564ebd3 P 398c3c8f3195492c900439ad5e1931bb [term 2 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 21:28:09.263476 14502 consensus_queue.cc:1035] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:09.272976 7482 raft_consensus.cc:2932] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:09.323392 4704 consensus_queue.cc:237] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:28:09.326678 7482 raft_consensus.cc:1261] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:09.326781 4574 raft_consensus.cc:1261] T fd610e340cd3409397d8f23b623b4074 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:28:09.327057 4639 consensus_peers.cc:485] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd610e340cd3409397d8f23b623b4074. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:09.327387 12351 consensus_queue.cc:1035] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:09.327661 14292 consensus_queue.cc:1035] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:09.331727 14292 raft_consensus.cc:2932] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:09.334708 7482 raft_consensus.cc:2932] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:09.339398 4574 raft_consensus.cc:2932] T fd610e340cd3409397d8f23b623b4074 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:09.347702 4298 catalog_manager.cc:5484] T fd610e340cd3409397d8f23b623b4074 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:09.438349 7618 raft_consensus.cc:1261] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 21:28:09.439069 14502 consensus_queue.cc:1035] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:09.476614 4574 raft_consensus.cc:1261] T 29235795acc147f68d2ccce93564ebd3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 2 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 21:28:09.477458 14502 consensus_queue.cc:1035] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:09.706355 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:09.728941 7462 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 21:28:09.754881 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:09.755288 7194 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 21:28:09.764895 4686 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:09.795150 4817 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 21:28:09.800290 14684 ts_tablet_manager.cc:902] T fb2cc722452f4d9798353d1bfae4bd4f P 73a1f2ca1f544de9b0e3b37015f85328: Initiating tablet copy from peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:28:09.838682 14689 ts_tablet_manager.cc:902] T fd610e340cd3409397d8f23b623b4074 P 71e1375b799b439fa3c83ca763c039c1: Initiating tablet copy from peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:28:09.858877 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:09.867235 14684 tablet_copy_client.cc:308] T fb2cc722452f4d9798353d1bfae4bd4f P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.65:42781
I20240502 21:28:09.868113 4463 tablet_copy_service.cc:140] P 71e1375b799b439fa3c83ca763c039c1: Received BeginTabletCopySession request for tablet fb2cc722452f4d9798353d1bfae4bd4f from peer 73a1f2ca1f544de9b0e3b37015f85328 ({username='slave'} at 127.29.82.68:58067)
I20240502 21:28:09.868278 4463 tablet_copy_service.cc:161] P 71e1375b799b439fa3c83ca763c039c1: Beginning new tablet copy session on tablet fb2cc722452f4d9798353d1bfae4bd4f from peer 73a1f2ca1f544de9b0e3b37015f85328 at {username='slave'} at 127.29.82.68:58067: session id = 73a1f2ca1f544de9b0e3b37015f85328-fb2cc722452f4d9798353d1bfae4bd4f
I20240502 21:28:09.871243 4463 tablet_copy_source_session.cc:205] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:09.875353 14684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb2cc722452f4d9798353d1bfae4bd4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:09.899631 14684 tablet_copy_client.cc:783] T fb2cc722452f4d9798353d1bfae4bd4f P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:09.900084 14684 tablet_copy_client.cc:647] T fb2cc722452f4d9798353d1bfae4bd4f P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:09.912454 14684 tablet_copy_client.cc:523] T fb2cc722452f4d9798353d1bfae4bd4f P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:09.915284 14684 tablet_bootstrap.cc:492] T fb2cc722452f4d9798353d1bfae4bd4f P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:28:09.938028 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:09.940496 14689 tablet_copy_client.cc:308] T fd610e340cd3409397d8f23b623b4074 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.67:34537
I20240502 21:28:09.941219 4727 tablet_copy_service.cc:140] P 398c3c8f3195492c900439ad5e1931bb: Received BeginTabletCopySession request for tablet fd610e340cd3409397d8f23b623b4074 from peer 71e1375b799b439fa3c83ca763c039c1 ({username='slave'} at 127.29.82.65:44561)
I20240502 21:28:09.941362 4727 tablet_copy_service.cc:161] P 398c3c8f3195492c900439ad5e1931bb: Beginning new tablet copy session on tablet fd610e340cd3409397d8f23b623b4074 from peer 71e1375b799b439fa3c83ca763c039c1 at {username='slave'} at 127.29.82.65:44561: session id = 71e1375b799b439fa3c83ca763c039c1-fd610e340cd3409397d8f23b623b4074
I20240502 21:28:09.944228 4727 tablet_copy_source_session.cc:205] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:09.945812 14689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd610e340cd3409397d8f23b623b4074. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:09.982040 14684 tablet_bootstrap.cc:492] T fb2cc722452f4d9798353d1bfae4bd4f P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:09.987288 14684 tablet_bootstrap.cc:492] T fb2cc722452f4d9798353d1bfae4bd4f P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap complete.
I20240502 21:28:09.987803 14684 ts_tablet_manager.cc:1367] T fb2cc722452f4d9798353d1bfae4bd4f P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.073s user 0.016s sys 0.035s
I20240502 21:28:09.990522 14684 raft_consensus.cc:348] T fb2cc722452f4d9798353d1bfae4bd4f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:28:09.992293 14684 raft_consensus.cc:729] T fb2cc722452f4d9798353d1bfae4bd4f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:28:09.994479 14684 consensus_queue.cc:260] T fb2cc722452f4d9798353d1bfae4bd4f P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:28:09.995240 14684 ts_tablet_manager.cc:1398] T fb2cc722452f4d9798353d1bfae4bd4f P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 21:28:09.997073 4463 tablet_copy_service.cc:342] P 71e1375b799b439fa3c83ca763c039c1: Request end of tablet copy session 73a1f2ca1f544de9b0e3b37015f85328-fb2cc722452f4d9798353d1bfae4bd4f received from {username='slave'} at 127.29.82.68:58067
I20240502 21:28:09.997198 4463 tablet_copy_service.cc:433] P 71e1375b799b439fa3c83ca763c039c1: ending tablet copy session 73a1f2ca1f544de9b0e3b37015f85328-fb2cc722452f4d9798353d1bfae4bd4f on tablet fb2cc722452f4d9798353d1bfae4bd4f with peer 73a1f2ca1f544de9b0e3b37015f85328
I20240502 21:28:10.009466 14689 tablet_copy_client.cc:783] T fd610e340cd3409397d8f23b623b4074 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:10.009717 14689 tablet_copy_client.cc:647] T fd610e340cd3409397d8f23b623b4074 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:10.029635 14689 tablet_copy_client.cc:523] T fd610e340cd3409397d8f23b623b4074 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:10.040381 14689 tablet_bootstrap.cc:492] T fd610e340cd3409397d8f23b623b4074 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:28:10.061643 14689 tablet_bootstrap.cc:492] T fd610e340cd3409397d8f23b623b4074 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:10.062059 14689 tablet_bootstrap.cc:492] T fd610e340cd3409397d8f23b623b4074 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap complete.
I20240502 21:28:10.062315 14689 ts_tablet_manager.cc:1367] T fd610e340cd3409397d8f23b623b4074 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.001s
I20240502 21:28:10.063062 14689 raft_consensus.cc:348] T fd610e340cd3409397d8f23b623b4074 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:28:10.063295 14689 raft_consensus.cc:729] T fd610e340cd3409397d8f23b623b4074 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:28:10.072983 14689 consensus_queue.cc:260] T fd610e340cd3409397d8f23b623b4074 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:28:10.073753 14689 ts_tablet_manager.cc:1398] T fd610e340cd3409397d8f23b623b4074 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.011s user 0.002s sys 0.003s
I20240502 21:28:10.078783 4727 tablet_copy_service.cc:342] P 398c3c8f3195492c900439ad5e1931bb: Request end of tablet copy session 71e1375b799b439fa3c83ca763c039c1-fd610e340cd3409397d8f23b623b4074 received from {username='slave'} at 127.29.82.65:44561
I20240502 21:28:10.078919 4727 tablet_copy_service.cc:433] P 398c3c8f3195492c900439ad5e1931bb: ending tablet copy session 71e1375b799b439fa3c83ca763c039c1-fd610e340cd3409397d8f23b623b4074 on tablet fd610e340cd3409397d8f23b623b4074 with peer 71e1375b799b439fa3c83ca763c039c1
I20240502 21:28:10.220208 4838 raft_consensus.cc:1203] T fb2cc722452f4d9798353d1bfae4bd4f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:10.288553 4442 raft_consensus.cc:1203] T fd610e340cd3409397d8f23b623b4074 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:10.517040 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:10.543630 4686 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:10.557974 7462 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 21:28:10.592464 7194 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 21:28:10.612547 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:10.668037 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:10.701915 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:10.704223 4817 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 21:28:10.918450 4442 consensus_queue.cc:237] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:28:10.920665 4838 raft_consensus.cc:1261] T fb2cc722452f4d9798353d1bfae4bd4f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:10.920850 7483 raft_consensus.cc:1261] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:10.921520 14599 consensus_queue.cc:1035] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:10.921825 14602 consensus_queue.cc:1035] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:10.926021 14599 raft_consensus.cc:2932] T fb2cc722452f4d9798353d1bfae4bd4f P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:10.928967 4839 raft_consensus.cc:2932] T fb2cc722452f4d9798353d1bfae4bd4f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:10.930073 7482 raft_consensus.cc:2932] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:10.938241 4297 catalog_manager.cc:5484] T fb2cc722452f4d9798353d1bfae4bd4f P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:10.948889 7328 tablet_service.cc:1514] Processing DeleteTablet for tablet fb2cc722452f4d9798353d1bfae4bd4f with delete_type TABLET_DATA_TOMBSTONED (TS d0456c5e55a542128688904e003d403c not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47078
I20240502 21:28:10.949599 14795 tablet_replica.cc:330] T fb2cc722452f4d9798353d1bfae4bd4f P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:28:10.949791 14795 raft_consensus.cc:2229] T fb2cc722452f4d9798353d1bfae4bd4f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:10.949980 14795 raft_consensus.cc:2258] T fb2cc722452f4d9798353d1bfae4bd4f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:10.951058 14795 ts_tablet_manager.cc:1875] T fb2cc722452f4d9798353d1bfae4bd4f P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:10.957386 14795 ts_tablet_manager.cc:1888] T fb2cc722452f4d9798353d1bfae4bd4f P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:10.957540 14795 log.cc:1196] T fb2cc722452f4d9798353d1bfae4bd4f P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/fb2cc722452f4d9798353d1bfae4bd4f
I20240502 21:28:10.958477 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet fb2cc722452f4d9798353d1bfae4bd4f (table rebalancer_extra_table_44 [id=0953e644d4b143a9b5a77699e921c8de]) successfully deleted
I20240502 21:28:11.262341 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:11.282725 7462 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 21:28:11.295025 4686 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:11.306385 7194 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 21:28:11.336970 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 21:28:11.410531 4639 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.68:42449, user_credentials={real_user=slave}} blocked reactor thread for 57510.7us
I20240502 21:28:11.430442 4817 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 21:28:11.446205 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:11.483800 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:11.617599 7212 consensus_queue.cc:237] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:11.621204 7350 raft_consensus.cc:1261] T 1d3be8c5eaac423db11b282739cd1d40 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:11.621788 14846 consensus_queue.cc:1035] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:11.622262 4703 raft_consensus.cc:1261] T 1d3be8c5eaac423db11b282739cd1d40 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:11.622792 14846 consensus_queue.cc:1035] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:11.625212 13938 raft_consensus.cc:2932] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:11.625813 7350 raft_consensus.cc:2932] T 1d3be8c5eaac423db11b282739cd1d40 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:11.632282 4704 raft_consensus.cc:2932] T 1d3be8c5eaac423db11b282739cd1d40 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:11.632442 4296 catalog_manager.cc:5484] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:11.659286 7462 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d3be8c5eaac423db11b282739cd1d40 with delete_type TABLET_DATA_TOMBSTONED (TS 31a72286397c4edc9a63eca3412be168 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:37182
I20240502 21:28:11.659946 14883 tablet_replica.cc:330] T 1d3be8c5eaac423db11b282739cd1d40 P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:28:11.660127 14883 raft_consensus.cc:2229] T 1d3be8c5eaac423db11b282739cd1d40 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:11.660275 14883 raft_consensus.cc:2258] T 1d3be8c5eaac423db11b282739cd1d40 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:11.660789 14883 ts_tablet_manager.cc:1875] T 1d3be8c5eaac423db11b282739cd1d40 P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:11.665733 14883 ts_tablet_manager.cc:1888] T 1d3be8c5eaac423db11b282739cd1d40 P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:11.665920 14883 log.cc:1196] T 1d3be8c5eaac423db11b282739cd1d40 P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/1d3be8c5eaac423db11b282739cd1d40
I20240502 21:28:11.666756 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet 1d3be8c5eaac423db11b282739cd1d40 (table rebalancer_extra_table_0 [id=ab7a7a9b6e69455484db6423f42c7c06]) successfully deleted
I20240502 21:28:12.105783 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:12.114409 4686 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:12.126595 7194 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 21:28:12.142642 7462 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 21:28:12.153838 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:12.191673 4817 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 21:28:12.218786 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:12.219816 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:12.421329 7350 consensus_queue.cc:237] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:28:12.423638 4440 raft_consensus.cc:1261] T b7c60d990751440392ab6e5feb0d8498 P 71e1375b799b439fa3c83ca763c039c1 [term 2 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:28:12.423671 4838 raft_consensus.cc:1261] T b7c60d990751440392ab6e5feb0d8498 P 73a1f2ca1f544de9b0e3b37015f85328 [term 2 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:28:12.424404 14732 consensus_queue.cc:1035] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:28:12.425515 14212 consensus_queue.cc:1035] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:28:12.429787 15006 raft_consensus.cc:2932] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:12.430325 4838 raft_consensus.cc:2932] T b7c60d990751440392ab6e5feb0d8498 P 73a1f2ca1f544de9b0e3b37015f85328 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:12.430738 4442 raft_consensus.cc:2932] T b7c60d990751440392ab6e5feb0d8498 P 71e1375b799b439fa3c83ca763c039c1 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:12.436389 4300 catalog_manager.cc:5484] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c reported cstate change: config changed from index 4 to 6, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New cstate: current_term: 2 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:12.458941 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet b7c60d990751440392ab6e5feb0d8498 with delete_type TABLET_DATA_TOMBSTONED (TS 398c3c8f3195492c900439ad5e1931bb not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44764
I20240502 21:28:12.464632 15015 tablet_replica.cc:330] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:28:12.465442 15015 raft_consensus.cc:2229] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:12.466032 15015 raft_consensus.cc:2258] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:12.466686 15015 ts_tablet_manager.cc:1875] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:12.477794 15015 ts_tablet_manager.cc:1888] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 21:28:12.477936 15015 log.cc:1196] T b7c60d990751440392ab6e5feb0d8498 P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/b7c60d990751440392ab6e5feb0d8498
I20240502 21:28:12.478658 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet b7c60d990751440392ab6e5feb0d8498 (table rebalance_test_table_19 [id=879d91ac491a4b2dbed04af44dc5e8de]) successfully deleted
I20240502 21:28:12.783908 7462 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 21:28:12.795007 4686 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:12.819063 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:12.872709 7194 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 21:28:12.926564 4817 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 21:28:12.932451 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:12.984083 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:13.043815 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:13.149159 4839 consensus_queue.cc:237] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:28:13.151018 7484 raft_consensus.cc:1261] T a5d47492f64f49d584246fc5007281aa P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:28:13.151037 7617 raft_consensus.cc:1261] T a5d47492f64f49d584246fc5007281aa P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:28:13.151625 14827 consensus_queue.cc:1035] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:13.152048 14944 consensus_queue.cc:1035] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:13.155380 14944 raft_consensus.cc:2932] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:13.156034 7617 raft_consensus.cc:2932] T a5d47492f64f49d584246fc5007281aa P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:13.160111 7484 raft_consensus.cc:2932] T a5d47492f64f49d584246fc5007281aa P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:13.167552 4291 catalog_manager.cc:5484] T a5d47492f64f49d584246fc5007281aa P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index 2 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New cstate: current_term: 2 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:13.201658 7194 tablet_service.cc:1514] Processing DeleteTablet for tablet a5d47492f64f49d584246fc5007281aa with delete_type TABLET_DATA_TOMBSTONED (TS 39981a8dfe94462b9a73ba489af4c714 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44652
I20240502 21:28:13.202368 15124 tablet_replica.cc:330] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:28:13.202673 15124 raft_consensus.cc:2229] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:13.202880 15124 raft_consensus.cc:2258] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:13.203696 15124 ts_tablet_manager.cc:1875] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:13.210573 15124 ts_tablet_manager.cc:1888] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240502 21:28:13.210748 15124 log.cc:1196] T a5d47492f64f49d584246fc5007281aa P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/a5d47492f64f49d584246fc5007281aa
I20240502 21:28:13.211390 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet a5d47492f64f49d584246fc5007281aa (table rebalancer_extra_table_30 [id=b87f68aedb34492aade9e648ad016b2c]) successfully deleted
I20240502 21:28:13.665634 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:13.677496 7462 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 21:28:13.706370 4686 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:13.725692 7194 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 21:28:13.775391 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:13.796141 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:13.816584 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 21:28:13.831884 7551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.69:33193, user_credentials={real_user=slave}} blocked reactor thread for 127466us
I20240502 21:28:13.847790 4817 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 21:28:14.065359 7484 consensus_queue.cc:237] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:28:14.067596 4575 raft_consensus.cc:1261] T 29235795acc147f68d2ccce93564ebd3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 2 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:28:14.067833 4703 raft_consensus.cc:1261] T 29235795acc147f68d2ccce93564ebd3 P 398c3c8f3195492c900439ad5e1931bb [term 2 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:28:14.068245 15112 consensus_queue.cc:1035] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:14.069001 15112 consensus_queue.cc:1035] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:14.072463 15112 raft_consensus.cc:2932] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:14.077062 4703 raft_consensus.cc:2932] T 29235795acc147f68d2ccce93564ebd3 P 398c3c8f3195492c900439ad5e1931bb [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:14.080762 4574 raft_consensus.cc:2932] T 29235795acc147f68d2ccce93564ebd3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:14.090806 4293 catalog_manager.cc:5484] T 29235795acc147f68d2ccce93564ebd3 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: config changed from index 2 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New cstate: current_term: 2 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:14.124071 7597 tablet_service.cc:1514] Processing DeleteTablet for tablet 29235795acc147f68d2ccce93564ebd3 with delete_type TABLET_DATA_TOMBSTONED (TS 0ddd486109f5434cace13d60d633ac04 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44444
I20240502 21:28:14.125813 15248 tablet_replica.cc:330] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04: stopping tablet replica
I20240502 21:28:14.126039 15248 raft_consensus.cc:2229] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:14.126485 15248 raft_consensus.cc:2258] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:14.127220 15248 ts_tablet_manager.cc:1875] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:14.132616 15248 ts_tablet_manager.cc:1888] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240502 21:28:14.132800 15248 log.cc:1196] T 29235795acc147f68d2ccce93564ebd3 P 0ddd486109f5434cace13d60d633ac04: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/wals/29235795acc147f68d2ccce93564ebd3
I20240502 21:28:14.133476 4285 catalog_manager.cc:4830] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): tablet 29235795acc147f68d2ccce93564ebd3 (table rebalancer_extra_table_28 [id=37d345f972f34b419de6c03d44f832d8]) successfully deleted
I20240502 21:28:14.156627 7214 consensus_queue.cc:237] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:28:14.158998 7617 raft_consensus.cc:1261] T be423032bc4b459b8e8e63a2379e37ac P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:14.159785 7349 raft_consensus.cc:1261] T be423032bc4b459b8e8e63a2379e37ac P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:14.160234 14846 consensus_queue.cc:1035] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:14.160782 14846 consensus_queue.cc:1035] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:28:14.162214 7148 consensus_peers.cc:485] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714 -> Peer 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): Couldn't send request to peer 31a72286397c4edc9a63eca3412be168. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: be423032bc4b459b8e8e63a2379e37ac. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:14.174312 13938 raft_consensus.cc:2932] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:14.175050 7617 raft_consensus.cc:2932] T be423032bc4b459b8e8e63a2379e37ac P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:14.184458 4299 catalog_manager.cc:5484] T be423032bc4b459b8e8e63a2379e37ac P 0ddd486109f5434cace13d60d633ac04 reported cstate change: config changed from index -1 to 2, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:14.185359 7349 raft_consensus.cc:2932] T be423032bc4b459b8e8e63a2379e37ac P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:14.232878 7214 consensus_queue.cc:237] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:14.235702 4574 raft_consensus.cc:1261] T 42edb2b3ff094c71872fde7949b61b3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:14.235730 4442 raft_consensus.cc:1261] T 42edb2b3ff094c71872fde7949b61b3b P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:14.236378 15262 consensus_queue.cc:1035] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:14.236666 14846 consensus_queue.cc:1035] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:28:14.237322 7151 consensus_peers.cc:485] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42edb2b3ff094c71872fde7949b61b3b. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:14.252825 14846 raft_consensus.cc:2932] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:14.254858 4574 raft_consensus.cc:2932] T 42edb2b3ff094c71872fde7949b61b3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:14.258167 4440 raft_consensus.cc:2932] T 42edb2b3ff094c71872fde7949b61b3b P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:14.264633 4292 catalog_manager.cc:5484] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:14.326279 7214 consensus_queue.cc:237] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:28:14.328631 4703 raft_consensus.cc:1261] T 831d24800da54da9b22f9e1bca474755 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:14.328843 4838 raft_consensus.cc:1261] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:14.329362 13938 consensus_queue.cc:1035] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:14.329910 13938 consensus_queue.cc:1035] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:28:14.330142 7150 consensus_peers.cc:485] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 -> Peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Couldn't send request to peer d0456c5e55a542128688904e003d403c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 831d24800da54da9b22f9e1bca474755. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:14.334102 15262 raft_consensus.cc:2932] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:14.336341 4703 raft_consensus.cc:2932] T 831d24800da54da9b22f9e1bca474755 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:14.341768 4838 raft_consensus.cc:2932] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:14.343951 4299 catalog_manager.cc:5484] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: config changed from index -1 to 2, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:14.401706 4838 consensus_queue.cc:237] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:28:14.409725 4703 raft_consensus.cc:1261] T 104c676bff964b8b8566b2200ce238c7 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:28:14.409881 4574 raft_consensus.cc:1261] T 104c676bff964b8b8566b2200ce238c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:28:14.410391 4771 consensus_peers.cc:485] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): Couldn't send request to peer 0ddd486109f5434cace13d60d633ac04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 104c676bff964b8b8566b2200ce238c7. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:14.411160 14944 consensus_queue.cc:1035] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:14.411662 14827 consensus_queue.cc:1035] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:14.415205 14827 raft_consensus.cc:2932] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:14.418439 4703 raft_consensus.cc:2932] T 104c676bff964b8b8566b2200ce238c7 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:14.418946 4574 raft_consensus.cc:2932] T 104c676bff964b8b8566b2200ce238c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:14.421762 4293 catalog_manager.cc:5484] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:14.478143 7618 consensus_queue.cc:237] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:28:14.481178 7214 raft_consensus.cc:1261] T cfceb76b56924833bd7e6b3901573ee7 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:14.481528 4704 raft_consensus.cc:1261] T cfceb76b56924833bd7e6b3901573ee7 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:28:14.481707 7549 consensus_peers.cc:485] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cfceb76b56924833bd7e6b3901573ee7. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:14.482060 14562 consensus_queue.cc:1035] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:14.482765 14562 consensus_queue.cc:1035] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:14.487295 14562 raft_consensus.cc:2932] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:14.487994 7214 raft_consensus.cc:2932] T cfceb76b56924833bd7e6b3901573ee7 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:14.491628 4704 raft_consensus.cc:2932] T cfceb76b56924833bd7e6b3901573ee7 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:14.495754 4292 catalog_manager.cc:5484] T cfceb76b56924833bd7e6b3901573ee7 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:14.533779 7214 consensus_queue.cc:237] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
W20240502 21:28:14.536283 7150 consensus_peers.cc:485] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 -> Peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Couldn't send request to peer d0456c5e55a542128688904e003d403c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a507a60fb2094bd3b9dc24bec06db867. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:14.536490 4838 raft_consensus.cc:1261] T a507a60fb2094bd3b9dc24bec06db867 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:14.536703 7484 raft_consensus.cc:1261] T a507a60fb2094bd3b9dc24bec06db867 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:14.537247 13938 consensus_queue.cc:1035] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:14.537544 14846 consensus_queue.cc:1035] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:14.540164 14846 raft_consensus.cc:2932] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:14.543434 4838 raft_consensus.cc:2932] T a507a60fb2094bd3b9dc24bec06db867 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:14.549551 7484 raft_consensus.cc:2932] T a507a60fb2094bd3b9dc24bec06db867 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:14.553530 4292 catalog_manager.cc:5484] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: config changed from index -1 to 2, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:14.590090 4703 consensus_queue.cc:237] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
W20240502 21:28:14.593295 4640 consensus_peers.cc:485] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb -> Peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): Couldn't send request to peer 0ddd486109f5434cace13d60d633ac04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 38ce97f9f9ac4398a16958931ef7262f. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:14.593888 4574 raft_consensus.cc:1261] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:28:14.594339 4443 raft_consensus.cc:1261] T 38ce97f9f9ac4398a16958931ef7262f P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:28:14.594374 15195 consensus_queue.cc:1035] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:14.596982 15195 consensus_queue.cc:1035] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:14.599762 15381 raft_consensus.cc:2932] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:14.599987 4443 raft_consensus.cc:2932] T 38ce97f9f9ac4398a16958931ef7262f P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:14.603152 4574 raft_consensus.cc:2932] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:14.606470 4299 catalog_manager.cc:5484] T 38ce97f9f9ac4398a16958931ef7262f P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:14.649713 15395 ts_tablet_manager.cc:902] T be423032bc4b459b8e8e63a2379e37ac P 31a72286397c4edc9a63eca3412be168: Initiating tablet copy from peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:28:14.651434 15395 tablet_copy_client.cc:308] T be423032bc4b459b8e8e63a2379e37ac P 31a72286397c4edc9a63eca3412be168: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.69:33193
I20240502 21:28:14.652262 7236 tablet_copy_service.cc:140] P 39981a8dfe94462b9a73ba489af4c714: Received BeginTabletCopySession request for tablet be423032bc4b459b8e8e63a2379e37ac from peer 31a72286397c4edc9a63eca3412be168 ({username='slave'} at 127.29.82.71:46939)
I20240502 21:28:14.652459 7236 tablet_copy_service.cc:161] P 39981a8dfe94462b9a73ba489af4c714: Beginning new tablet copy session on tablet be423032bc4b459b8e8e63a2379e37ac from peer 31a72286397c4edc9a63eca3412be168 at {username='slave'} at 127.29.82.71:46939: session id = 31a72286397c4edc9a63eca3412be168-be423032bc4b459b8e8e63a2379e37ac
I20240502 21:28:14.655297 7236 tablet_copy_source_session.cc:205] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:14.658205 15395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be423032bc4b459b8e8e63a2379e37ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:14.664815 15395 tablet_copy_client.cc:783] T be423032bc4b459b8e8e63a2379e37ac P 31a72286397c4edc9a63eca3412be168: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:14.665135 15395 tablet_copy_client.cc:647] T be423032bc4b459b8e8e63a2379e37ac P 31a72286397c4edc9a63eca3412be168: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:14.666110 7484 consensus_queue.cc:237] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:28:14.666608 15395 tablet_copy_client.cc:523] T be423032bc4b459b8e8e63a2379e37ac P 31a72286397c4edc9a63eca3412be168: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:14.669894 15395 tablet_bootstrap.cc:492] T be423032bc4b459b8e8e63a2379e37ac P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
W20240502 21:28:14.671030 7420 consensus_peers.cc:485] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f800bfd495a94f2a952bc57a348304d0. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:14.671032 7214 raft_consensus.cc:1261] T f800bfd495a94f2a952bc57a348304d0 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:14.671638 15112 consensus_queue.cc:1035] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:14.672392 7617 raft_consensus.cc:1261] T f800bfd495a94f2a952bc57a348304d0 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:14.672979 15112 consensus_queue.cc:1035] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:14.675963 15239 raft_consensus.cc:2932] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:14.677022 7617 raft_consensus.cc:2932] T f800bfd495a94f2a952bc57a348304d0 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:14.677532 15395 tablet_bootstrap.cc:492] T be423032bc4b459b8e8e63a2379e37ac P 31a72286397c4edc9a63eca3412be168: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:14.677882 15395 tablet_bootstrap.cc:492] T be423032bc4b459b8e8e63a2379e37ac P 31a72286397c4edc9a63eca3412be168: Bootstrap complete.
I20240502 21:28:14.678100 15395 ts_tablet_manager.cc:1367] T be423032bc4b459b8e8e63a2379e37ac P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240502 21:28:14.678752 15395 raft_consensus.cc:348] T be423032bc4b459b8e8e63a2379e37ac P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:28:14.678941 15395 raft_consensus.cc:729] T be423032bc4b459b8e8e63a2379e37ac P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:28:14.679139 15395 consensus_queue.cc:260] T be423032bc4b459b8e8e63a2379e37ac P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:28:14.679566 15395 ts_tablet_manager.cc:1398] T be423032bc4b459b8e8e63a2379e37ac P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:28:14.682341 7236 tablet_copy_service.cc:342] P 39981a8dfe94462b9a73ba489af4c714: Request end of tablet copy session 31a72286397c4edc9a63eca3412be168-be423032bc4b459b8e8e63a2379e37ac received from {username='slave'} at 127.29.82.71:46939
I20240502 21:28:14.682463 7236 tablet_copy_service.cc:433] P 39981a8dfe94462b9a73ba489af4c714: ending tablet copy session 31a72286397c4edc9a63eca3412be168-be423032bc4b459b8e8e63a2379e37ac on tablet be423032bc4b459b8e8e63a2379e37ac with peer 31a72286397c4edc9a63eca3412be168
I20240502 21:28:14.691812 4293 catalog_manager.cc:5484] T f800bfd495a94f2a952bc57a348304d0 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:14.692816 7214 raft_consensus.cc:2932] T f800bfd495a94f2a952bc57a348304d0 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:14.726315 15419 ts_tablet_manager.cc:902] T 831d24800da54da9b22f9e1bca474755 P d0456c5e55a542128688904e003d403c: Initiating tablet copy from peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:28:14.734813 15419 tablet_copy_client.cc:308] T 831d24800da54da9b22f9e1bca474755 P d0456c5e55a542128688904e003d403c: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.69:33193
I20240502 21:28:14.735805 7236 tablet_copy_service.cc:140] P 39981a8dfe94462b9a73ba489af4c714: Received BeginTabletCopySession request for tablet 831d24800da54da9b22f9e1bca474755 from peer d0456c5e55a542128688904e003d403c ({username='slave'} at 127.29.82.70:60977)
I20240502 21:28:14.735976 7236 tablet_copy_service.cc:161] P 39981a8dfe94462b9a73ba489af4c714: Beginning new tablet copy session on tablet 831d24800da54da9b22f9e1bca474755 from peer d0456c5e55a542128688904e003d403c at {username='slave'} at 127.29.82.70:60977: session id = d0456c5e55a542128688904e003d403c-831d24800da54da9b22f9e1bca474755
I20240502 21:28:14.738396 7236 tablet_copy_source_session.cc:205] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:14.739542 15419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 831d24800da54da9b22f9e1bca474755. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:14.747882 15419 tablet_copy_client.cc:783] T 831d24800da54da9b22f9e1bca474755 P d0456c5e55a542128688904e003d403c: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:14.748111 15419 tablet_copy_client.cc:647] T 831d24800da54da9b22f9e1bca474755 P d0456c5e55a542128688904e003d403c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:14.750346 15419 tablet_copy_client.cc:523] T 831d24800da54da9b22f9e1bca474755 P d0456c5e55a542128688904e003d403c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:14.753978 7617 consensus_queue.cc:237] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:28:14.756016 15419 tablet_bootstrap.cc:492] T 831d24800da54da9b22f9e1bca474755 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:28:14.756574 7349 raft_consensus.cc:1261] T d558bcaf20bd46548936d5fb946d7710 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:14.757134 15341 consensus_queue.cc:1035] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:14.757402 7484 raft_consensus.cc:1261] T d558bcaf20bd46548936d5fb946d7710 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:28:14.758020 7549 consensus_peers.cc:485] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d558bcaf20bd46548936d5fb946d7710. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:14.758697 15341 consensus_queue.cc:1035] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:14.841367 15432 ts_tablet_manager.cc:902] T 42edb2b3ff094c71872fde7949b61b3b P 398c3c8f3195492c900439ad5e1931bb: Initiating tablet copy from peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:28:14.859205 15419 tablet_bootstrap.cc:492] T 831d24800da54da9b22f9e1bca474755 P d0456c5e55a542128688904e003d403c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:14.859763 15419 tablet_bootstrap.cc:492] T 831d24800da54da9b22f9e1bca474755 P d0456c5e55a542128688904e003d403c: Bootstrap complete.
I20240502 21:28:14.860095 15419 ts_tablet_manager.cc:1367] T 831d24800da54da9b22f9e1bca474755 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.104s user 0.039s sys 0.049s
I20240502 21:28:14.860906 15419 raft_consensus.cc:348] T 831d24800da54da9b22f9e1bca474755 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:28:14.861223 15419 raft_consensus.cc:729] T 831d24800da54da9b22f9e1bca474755 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:28:14.861536 15419 consensus_queue.cc:260] T 831d24800da54da9b22f9e1bca474755 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:28:14.862182 15419 ts_tablet_manager.cc:1398] T 831d24800da54da9b22f9e1bca474755 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:28:14.862831 15341 raft_consensus.cc:2932] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:14.865911 7236 tablet_copy_service.cc:342] P 39981a8dfe94462b9a73ba489af4c714: Request end of tablet copy session d0456c5e55a542128688904e003d403c-831d24800da54da9b22f9e1bca474755 received from {username='slave'} at 127.29.82.70:60977
I20240502 21:28:14.866030 7236 tablet_copy_service.cc:433] P 39981a8dfe94462b9a73ba489af4c714: ending tablet copy session d0456c5e55a542128688904e003d403c-831d24800da54da9b22f9e1bca474755 on tablet 831d24800da54da9b22f9e1bca474755 with peer d0456c5e55a542128688904e003d403c
I20240502 21:28:14.869532 7482 raft_consensus.cc:2932] T d558bcaf20bd46548936d5fb946d7710 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:14.873601 7349 raft_consensus.cc:2932] T d558bcaf20bd46548936d5fb946d7710 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:14.877733 4299 catalog_manager.cc:5484] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:14.896559 15439 ts_tablet_manager.cc:902] T 104c676bff964b8b8566b2200ce238c7 P 0ddd486109f5434cace13d60d633ac04: Initiating tablet copy from peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:28:14.905092 15432 tablet_copy_client.cc:308] T 42edb2b3ff094c71872fde7949b61b3b P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.69:33193
I20240502 21:28:14.905727 7236 tablet_copy_service.cc:140] P 39981a8dfe94462b9a73ba489af4c714: Received BeginTabletCopySession request for tablet 42edb2b3ff094c71872fde7949b61b3b from peer 398c3c8f3195492c900439ad5e1931bb ({username='slave'} at 127.29.82.67:59759)
I20240502 21:28:14.905897 7236 tablet_copy_service.cc:161] P 39981a8dfe94462b9a73ba489af4c714: Beginning new tablet copy session on tablet 42edb2b3ff094c71872fde7949b61b3b from peer 398c3c8f3195492c900439ad5e1931bb at {username='slave'} at 127.29.82.67:59759: session id = 398c3c8f3195492c900439ad5e1931bb-42edb2b3ff094c71872fde7949b61b3b
I20240502 21:28:14.908984 7236 tablet_copy_source_session.cc:205] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:14.911880 15432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42edb2b3ff094c71872fde7949b61b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:14.929548 15432 tablet_copy_client.cc:783] T 42edb2b3ff094c71872fde7949b61b3b P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:14.929782 15432 tablet_copy_client.cc:647] T 42edb2b3ff094c71872fde7949b61b3b P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:14.931223 15432 tablet_copy_client.cc:523] T 42edb2b3ff094c71872fde7949b61b3b P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:14.935302 15446 ts_tablet_manager.cc:902] T cfceb76b56924833bd7e6b3901573ee7 P 73a1f2ca1f544de9b0e3b37015f85328: Initiating tablet copy from peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:28:14.949206 15432 tablet_bootstrap.cc:492] T 42edb2b3ff094c71872fde7949b61b3b P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:28:14.965090 15446 tablet_copy_client.cc:308] T cfceb76b56924833bd7e6b3901573ee7 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.72:41223
I20240502 21:28:14.965922 7638 tablet_copy_service.cc:140] P 0ddd486109f5434cace13d60d633ac04: Received BeginTabletCopySession request for tablet cfceb76b56924833bd7e6b3901573ee7 from peer 73a1f2ca1f544de9b0e3b37015f85328 ({username='slave'} at 127.29.82.68:47485)
I20240502 21:28:14.966102 7638 tablet_copy_service.cc:161] P 0ddd486109f5434cace13d60d633ac04: Beginning new tablet copy session on tablet cfceb76b56924833bd7e6b3901573ee7 from peer 73a1f2ca1f544de9b0e3b37015f85328 at {username='slave'} at 127.29.82.68:47485: session id = 73a1f2ca1f544de9b0e3b37015f85328-cfceb76b56924833bd7e6b3901573ee7
I20240502 21:28:14.969187 7638 tablet_copy_source_session.cc:205] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:14.970988 15446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfceb76b56924833bd7e6b3901573ee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:14.974459 15439 tablet_copy_client.cc:308] T 104c676bff964b8b8566b2200ce238c7 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.68:42449
I20240502 21:28:14.975073 4859 tablet_copy_service.cc:140] P 73a1f2ca1f544de9b0e3b37015f85328: Received BeginTabletCopySession request for tablet 104c676bff964b8b8566b2200ce238c7 from peer 0ddd486109f5434cace13d60d633ac04 ({username='slave'} at 127.29.82.72:48273)
I20240502 21:28:14.975222 4859 tablet_copy_service.cc:161] P 73a1f2ca1f544de9b0e3b37015f85328: Beginning new tablet copy session on tablet 104c676bff964b8b8566b2200ce238c7 from peer 0ddd486109f5434cace13d60d633ac04 at {username='slave'} at 127.29.82.72:48273: session id = 0ddd486109f5434cace13d60d633ac04-104c676bff964b8b8566b2200ce238c7
I20240502 21:28:14.978593 4859 tablet_copy_source_session.cc:205] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:14.980005 15446 tablet_copy_client.cc:783] T cfceb76b56924833bd7e6b3901573ee7 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:14.980909 15446 tablet_copy_client.cc:647] T cfceb76b56924833bd7e6b3901573ee7 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:14.981618 15439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 104c676bff964b8b8566b2200ce238c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:14.984113 15446 tablet_copy_client.cc:523] T cfceb76b56924833bd7e6b3901573ee7 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:14.987805 15432 tablet_bootstrap.cc:492] T 42edb2b3ff094c71872fde7949b61b3b P 398c3c8f3195492c900439ad5e1931bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:14.987972 15446 tablet_bootstrap.cc:492] T cfceb76b56924833bd7e6b3901573ee7 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:28:14.988185 15432 tablet_bootstrap.cc:492] T 42edb2b3ff094c71872fde7949b61b3b P 398c3c8f3195492c900439ad5e1931bb: Bootstrap complete.
I20240502 21:28:14.988452 15432 ts_tablet_manager.cc:1367] T 42edb2b3ff094c71872fde7949b61b3b P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.039s user 0.009s sys 0.000s
I20240502 21:28:14.989615 15432 raft_consensus.cc:348] T 42edb2b3ff094c71872fde7949b61b3b P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:14.991422 15432 raft_consensus.cc:729] T 42edb2b3ff094c71872fde7949b61b3b P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:28:14.991793 15432 consensus_queue.cc:260] T 42edb2b3ff094c71872fde7949b61b3b P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:14.992532 15432 ts_tablet_manager.cc:1398] T 42edb2b3ff094c71872fde7949b61b3b P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:28:14.993381 15439 tablet_copy_client.cc:783] T 104c676bff964b8b8566b2200ce238c7 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:14.993657 15439 tablet_copy_client.cc:647] T 104c676bff964b8b8566b2200ce238c7 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:14.994513 7236 tablet_copy_service.cc:342] P 39981a8dfe94462b9a73ba489af4c714: Request end of tablet copy session 398c3c8f3195492c900439ad5e1931bb-42edb2b3ff094c71872fde7949b61b3b received from {username='slave'} at 127.29.82.67:59759
I20240502 21:28:14.994635 7236 tablet_copy_service.cc:433] P 39981a8dfe94462b9a73ba489af4c714: ending tablet copy session 398c3c8f3195492c900439ad5e1931bb-42edb2b3ff094c71872fde7949b61b3b on tablet 42edb2b3ff094c71872fde7949b61b3b with peer 398c3c8f3195492c900439ad5e1931bb
I20240502 21:28:15.003031 15439 tablet_copy_client.cc:523] T 104c676bff964b8b8566b2200ce238c7 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:15.007612 15439 tablet_bootstrap.cc:492] T 104c676bff964b8b8566b2200ce238c7 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:28:15.008924 15446 tablet_bootstrap.cc:492] T cfceb76b56924833bd7e6b3901573ee7 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:15.009346 15446 tablet_bootstrap.cc:492] T cfceb76b56924833bd7e6b3901573ee7 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap complete.
I20240502 21:28:15.009645 15446 ts_tablet_manager.cc:1367] T cfceb76b56924833bd7e6b3901573ee7 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.008s
I20240502 21:28:15.010536 15446 raft_consensus.cc:348] T cfceb76b56924833bd7e6b3901573ee7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:28:15.010819 15446 raft_consensus.cc:729] T cfceb76b56924833bd7e6b3901573ee7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:28:15.011178 15446 consensus_queue.cc:260] T cfceb76b56924833bd7e6b3901573ee7 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:28:15.013298 15446 ts_tablet_manager.cc:1398] T cfceb76b56924833bd7e6b3901573ee7 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 21:28:15.019697 7638 tablet_copy_service.cc:342] P 0ddd486109f5434cace13d60d633ac04: Request end of tablet copy session 73a1f2ca1f544de9b0e3b37015f85328-cfceb76b56924833bd7e6b3901573ee7 received from {username='slave'} at 127.29.82.68:47485
I20240502 21:28:15.019811 7638 tablet_copy_service.cc:433] P 0ddd486109f5434cace13d60d633ac04: ending tablet copy session 73a1f2ca1f544de9b0e3b37015f85328-cfceb76b56924833bd7e6b3901573ee7 on tablet cfceb76b56924833bd7e6b3901573ee7 with peer 73a1f2ca1f544de9b0e3b37015f85328
I20240502 21:28:15.039994 15457 ts_tablet_manager.cc:902] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04: Initiating tablet copy from peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:28:15.081634 15457 tablet_copy_client.cc:308] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.67:34537
I20240502 21:28:15.082463 4727 tablet_copy_service.cc:140] P 398c3c8f3195492c900439ad5e1931bb: Received BeginTabletCopySession request for tablet 38ce97f9f9ac4398a16958931ef7262f from peer 0ddd486109f5434cace13d60d633ac04 ({username='slave'} at 127.29.82.72:33287)
I20240502 21:28:15.082607 4727 tablet_copy_service.cc:161] P 398c3c8f3195492c900439ad5e1931bb: Beginning new tablet copy session on tablet 38ce97f9f9ac4398a16958931ef7262f from peer 0ddd486109f5434cace13d60d633ac04 at {username='slave'} at 127.29.82.72:33287: session id = 0ddd486109f5434cace13d60d633ac04-38ce97f9f9ac4398a16958931ef7262f
I20240502 21:28:15.085462 4727 tablet_copy_source_session.cc:205] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:15.087045 15457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38ce97f9f9ac4398a16958931ef7262f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:15.094368 15439 tablet_bootstrap.cc:492] T 104c676bff964b8b8566b2200ce238c7 P 0ddd486109f5434cace13d60d633ac04: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:15.094775 15439 tablet_bootstrap.cc:492] T 104c676bff964b8b8566b2200ce238c7 P 0ddd486109f5434cace13d60d633ac04: Bootstrap complete.
I20240502 21:28:15.095032 15439 ts_tablet_manager.cc:1367] T 104c676bff964b8b8566b2200ce238c7 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.088s user 0.051s sys 0.007s
I20240502 21:28:15.095835 15439 raft_consensus.cc:348] T 104c676bff964b8b8566b2200ce238c7 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:28:15.096292 15439 raft_consensus.cc:729] T 104c676bff964b8b8566b2200ce238c7 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:28:15.096527 15439 consensus_queue.cc:260] T 104c676bff964b8b8566b2200ce238c7 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:28:15.098487 15439 ts_tablet_manager.cc:1398] T 104c676bff964b8b8566b2200ce238c7 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:28:15.099184 4859 tablet_copy_service.cc:342] P 73a1f2ca1f544de9b0e3b37015f85328: Request end of tablet copy session 0ddd486109f5434cace13d60d633ac04-104c676bff964b8b8566b2200ce238c7 received from {username='slave'} at 127.29.82.72:48273
I20240502 21:28:15.099298 4859 tablet_copy_service.cc:433] P 73a1f2ca1f544de9b0e3b37015f85328: ending tablet copy session 0ddd486109f5434cace13d60d633ac04-104c676bff964b8b8566b2200ce238c7 on tablet 104c676bff964b8b8566b2200ce238c7 with peer 0ddd486109f5434cace13d60d633ac04
I20240502 21:28:15.118583 15457 tablet_copy_client.cc:783] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:15.118886 15457 tablet_copy_client.cc:647] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:15.120621 15457 tablet_copy_client.cc:523] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:15.124639 15457 tablet_bootstrap.cc:492] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:28:15.143810 15419 ts_tablet_manager.cc:902] T a507a60fb2094bd3b9dc24bec06db867 P d0456c5e55a542128688904e003d403c: Initiating tablet copy from peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:28:15.145246 15419 tablet_copy_client.cc:308] T a507a60fb2094bd3b9dc24bec06db867 P d0456c5e55a542128688904e003d403c: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.69:33193
I20240502 21:28:15.146137 7236 tablet_copy_service.cc:140] P 39981a8dfe94462b9a73ba489af4c714: Received BeginTabletCopySession request for tablet a507a60fb2094bd3b9dc24bec06db867 from peer d0456c5e55a542128688904e003d403c ({username='slave'} at 127.29.82.70:60977)
I20240502 21:28:15.146291 7236 tablet_copy_service.cc:161] P 39981a8dfe94462b9a73ba489af4c714: Beginning new tablet copy session on tablet a507a60fb2094bd3b9dc24bec06db867 from peer d0456c5e55a542128688904e003d403c at {username='slave'} at 127.29.82.70:60977: session id = d0456c5e55a542128688904e003d403c-a507a60fb2094bd3b9dc24bec06db867
I20240502 21:28:15.149297 7236 tablet_copy_source_session.cc:205] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:15.153354 15419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a507a60fb2094bd3b9dc24bec06db867. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:15.161361 15419 tablet_copy_client.cc:783] T a507a60fb2094bd3b9dc24bec06db867 P d0456c5e55a542128688904e003d403c: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:15.161573 15419 tablet_copy_client.cc:647] T a507a60fb2094bd3b9dc24bec06db867 P d0456c5e55a542128688904e003d403c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:15.162963 15419 tablet_copy_client.cc:523] T a507a60fb2094bd3b9dc24bec06db867 P d0456c5e55a542128688904e003d403c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:15.166494 15419 tablet_bootstrap.cc:492] T a507a60fb2094bd3b9dc24bec06db867 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:28:15.171334 7482 raft_consensus.cc:1203] T be423032bc4b459b8e8e63a2379e37ac P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:15.175721 15457 tablet_bootstrap.cc:492] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:15.176113 15457 tablet_bootstrap.cc:492] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04: Bootstrap complete.
I20240502 21:28:15.176147 7350 raft_consensus.cc:1203] T 831d24800da54da9b22f9e1bca474755 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:15.176383 15457 ts_tablet_manager.cc:1367] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.052s user 0.043s sys 0.002s
I20240502 21:28:15.177064 15457 raft_consensus.cc:348] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:28:15.177278 15457 raft_consensus.cc:729] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:28:15.177511 15457 consensus_queue.cc:260] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:28:15.178167 15457 ts_tablet_manager.cc:1398] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:28:15.178997 15419 tablet_bootstrap.cc:492] T a507a60fb2094bd3b9dc24bec06db867 P d0456c5e55a542128688904e003d403c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:15.179399 15419 tablet_bootstrap.cc:492] T a507a60fb2094bd3b9dc24bec06db867 P d0456c5e55a542128688904e003d403c: Bootstrap complete.
I20240502 21:28:15.179678 15419 ts_tablet_manager.cc:1367] T a507a60fb2094bd3b9dc24bec06db867 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.009s
I20240502 21:28:15.179682 4727 tablet_copy_service.cc:342] P 398c3c8f3195492c900439ad5e1931bb: Request end of tablet copy session 0ddd486109f5434cace13d60d633ac04-38ce97f9f9ac4398a16958931ef7262f received from {username='slave'} at 127.29.82.72:33287
I20240502 21:28:15.179786 4727 tablet_copy_service.cc:433] P 398c3c8f3195492c900439ad5e1931bb: ending tablet copy session 0ddd486109f5434cace13d60d633ac04-38ce97f9f9ac4398a16958931ef7262f on tablet 38ce97f9f9ac4398a16958931ef7262f with peer 0ddd486109f5434cace13d60d633ac04
I20240502 21:28:15.180385 15419 raft_consensus.cc:348] T a507a60fb2094bd3b9dc24bec06db867 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:28:15.181080 15419 raft_consensus.cc:729] T a507a60fb2094bd3b9dc24bec06db867 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:28:15.181336 15419 consensus_queue.cc:260] T a507a60fb2094bd3b9dc24bec06db867 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:28:15.181963 15419 ts_tablet_manager.cc:1398] T a507a60fb2094bd3b9dc24bec06db867 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:28:15.185149 7236 tablet_copy_service.cc:342] P 39981a8dfe94462b9a73ba489af4c714: Request end of tablet copy session d0456c5e55a542128688904e003d403c-a507a60fb2094bd3b9dc24bec06db867 received from {username='slave'} at 127.29.82.70:60977
I20240502 21:28:15.185267 7236 tablet_copy_service.cc:433] P 39981a8dfe94462b9a73ba489af4c714: ending tablet copy session d0456c5e55a542128688904e003d403c-a507a60fb2094bd3b9dc24bec06db867 on tablet a507a60fb2094bd3b9dc24bec06db867 with peer d0456c5e55a542128688904e003d403c
I20240502 21:28:15.193383 7482 consensus_queue.cc:237] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:15.195999 7618 raft_consensus.cc:1261] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:15.196177 7350 raft_consensus.cc:1261] T 2ae670e3d8ae49f2b8c7df65b45b179e P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:15.196518 15239 consensus_queue.cc:1035] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:15.196797 15112 consensus_queue.cc:1035] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:28:15.197271 7420 consensus_peers.cc:485] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ae670e3d8ae49f2b8c7df65b45b179e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:15.199254 15239 raft_consensus.cc:2932] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:15.199923 7618 raft_consensus.cc:2932] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:15.205030 7350 raft_consensus.cc:2932] T 2ae670e3d8ae49f2b8c7df65b45b179e P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:15.206164 4299 catalog_manager.cc:5484] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 reported cstate change: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:15.229092 15493 ts_tablet_manager.cc:902] T f800bfd495a94f2a952bc57a348304d0 P 7a736a8ebb164ce0a051c0fccd76c3b8: Initiating tablet copy from peer 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:28:15.232414 15493 tablet_copy_client.cc:308] T f800bfd495a94f2a952bc57a348304d0 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.71:33913
I20240502 21:28:15.244153 7505 tablet_copy_service.cc:140] P 31a72286397c4edc9a63eca3412be168: Received BeginTabletCopySession request for tablet f800bfd495a94f2a952bc57a348304d0 from peer 7a736a8ebb164ce0a051c0fccd76c3b8 ({username='slave'} at 127.29.82.66:50655)
I20240502 21:28:15.244302 7505 tablet_copy_service.cc:161] P 31a72286397c4edc9a63eca3412be168: Beginning new tablet copy session on tablet f800bfd495a94f2a952bc57a348304d0 from peer 7a736a8ebb164ce0a051c0fccd76c3b8 at {username='slave'} at 127.29.82.66:50655: session id = 7a736a8ebb164ce0a051c0fccd76c3b8-f800bfd495a94f2a952bc57a348304d0
I20240502 21:28:15.245836 7214 consensus_queue.cc:237] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:28:15.248229 7618 raft_consensus.cc:1261] T 61e18d2b13644f6a99225c106c594914 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:28:15.248461 7148 consensus_peers.cc:485] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61e18d2b13644f6a99225c106c594914. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:15.248855 7350 raft_consensus.cc:1261] T 61e18d2b13644f6a99225c106c594914 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:15.248843 13938 consensus_queue.cc:1035] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:15.251771 13938 consensus_queue.cc:1035] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:15.255710 7505 tablet_copy_source_session.cc:205] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:15.257329 15493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f800bfd495a94f2a952bc57a348304d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:15.264880 15493 tablet_copy_client.cc:783] T f800bfd495a94f2a952bc57a348304d0 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:15.265094 15493 tablet_copy_client.cc:647] T f800bfd495a94f2a952bc57a348304d0 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:15.267824 15493 tablet_copy_client.cc:523] T f800bfd495a94f2a952bc57a348304d0 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:15.269299 4703 raft_consensus.cc:1203] T 42edb2b3ff094c71872fde7949b61b3b P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:15.271291 15493 tablet_bootstrap.cc:492] T f800bfd495a94f2a952bc57a348304d0 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:28:15.274695 15446 ts_tablet_manager.cc:902] T d558bcaf20bd46548936d5fb946d7710 P 73a1f2ca1f544de9b0e3b37015f85328: Initiating tablet copy from peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:28:15.277555 15262 raft_consensus.cc:2932] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:15.280498 7348 raft_consensus.cc:2932] T 61e18d2b13644f6a99225c106c594914 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:15.291044 7617 raft_consensus.cc:2932] T 61e18d2b13644f6a99225c106c594914 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:15.295465 15493 tablet_bootstrap.cc:492] T f800bfd495a94f2a952bc57a348304d0 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:15.295197 4292 catalog_manager.cc:5484] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:15.307965 15493 tablet_bootstrap.cc:492] T f800bfd495a94f2a952bc57a348304d0 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap complete.
I20240502 21:28:15.308338 15493 ts_tablet_manager.cc:1367] T f800bfd495a94f2a952bc57a348304d0 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.016s
I20240502 21:28:15.309260 15493 raft_consensus.cc:348] T f800bfd495a94f2a952bc57a348304d0 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:28:15.309587 15493 raft_consensus.cc:729] T f800bfd495a94f2a952bc57a348304d0 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:28:15.309854 15493 consensus_queue.cc:260] T f800bfd495a94f2a952bc57a348304d0 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:28:15.310528 15493 ts_tablet_manager.cc:1398] T f800bfd495a94f2a952bc57a348304d0 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:28:15.311419 7505 tablet_copy_service.cc:342] P 31a72286397c4edc9a63eca3412be168: Request end of tablet copy session 7a736a8ebb164ce0a051c0fccd76c3b8-f800bfd495a94f2a952bc57a348304d0 received from {username='slave'} at 127.29.82.66:50655
I20240502 21:28:15.311556 7505 tablet_copy_service.cc:433] P 31a72286397c4edc9a63eca3412be168: ending tablet copy session 7a736a8ebb164ce0a051c0fccd76c3b8-f800bfd495a94f2a952bc57a348304d0 on tablet f800bfd495a94f2a952bc57a348304d0 with peer 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:28:15.341652 15446 tablet_copy_client.cc:308] T d558bcaf20bd46548936d5fb946d7710 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.72:41223
I20240502 21:28:15.342335 7638 tablet_copy_service.cc:140] P 0ddd486109f5434cace13d60d633ac04: Received BeginTabletCopySession request for tablet d558bcaf20bd46548936d5fb946d7710 from peer 73a1f2ca1f544de9b0e3b37015f85328 ({username='slave'} at 127.29.82.68:47485)
I20240502 21:28:15.342491 7638 tablet_copy_service.cc:161] P 0ddd486109f5434cace13d60d633ac04: Beginning new tablet copy session on tablet d558bcaf20bd46548936d5fb946d7710 from peer 73a1f2ca1f544de9b0e3b37015f85328 at {username='slave'} at 127.29.82.68:47485: session id = 73a1f2ca1f544de9b0e3b37015f85328-d558bcaf20bd46548936d5fb946d7710
I20240502 21:28:15.345283 7638 tablet_copy_source_session.cc:205] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:15.346698 15446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d558bcaf20bd46548936d5fb946d7710. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:15.360607 15446 tablet_copy_client.cc:783] T d558bcaf20bd46548936d5fb946d7710 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:15.360837 15446 tablet_copy_client.cc:647] T d558bcaf20bd46548936d5fb946d7710 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:15.362877 15446 tablet_copy_client.cc:523] T d558bcaf20bd46548936d5fb946d7710 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:15.365789 15446 tablet_bootstrap.cc:492] T d558bcaf20bd46548936d5fb946d7710 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:28:15.419332 7617 raft_consensus.cc:1203] T 104c676bff964b8b8566b2200ce238c7 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:28:15.423170 4839 consensus_queue.cc:237] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:28:15.428386 7214 raft_consensus.cc:1261] T 93e0911f163f4629a4df0249c5eb8860 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:28:15.428712 4773 consensus_peers.cc:485] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 93e0911f163f4629a4df0249c5eb8860. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:15.429147 14827 consensus_queue.cc:1035] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:15.430318 7482 raft_consensus.cc:1261] T 93e0911f163f4629a4df0249c5eb8860 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:15.431946 14827 consensus_queue.cc:1035] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:15.437184 14827 raft_consensus.cc:2932] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:15.439481 7212 raft_consensus.cc:2932] T 93e0911f163f4629a4df0249c5eb8860 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:15.447494 4292 catalog_manager.cc:5484] T 93e0911f163f4629a4df0249c5eb8860 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:15.462466 15446 tablet_bootstrap.cc:492] T d558bcaf20bd46548936d5fb946d7710 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:15.462944 15446 tablet_bootstrap.cc:492] T d558bcaf20bd46548936d5fb946d7710 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap complete.
I20240502 21:28:15.463609 7482 raft_consensus.cc:2932] T 93e0911f163f4629a4df0249c5eb8860 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:15.463235 15446 ts_tablet_manager.cc:1367] T d558bcaf20bd46548936d5fb946d7710 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.098s user 0.029s sys 0.038s
I20240502 21:28:15.470824 4839 raft_consensus.cc:1203] T cfceb76b56924833bd7e6b3901573ee7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:15.470906 15446 raft_consensus.cc:348] T d558bcaf20bd46548936d5fb946d7710 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:28:15.471114 15446 raft_consensus.cc:729] T d558bcaf20bd46548936d5fb946d7710 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:28:15.472374 15446 consensus_queue.cc:260] T d558bcaf20bd46548936d5fb946d7710 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:28:15.473181 15446 ts_tablet_manager.cc:1398] T d558bcaf20bd46548936d5fb946d7710 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:28:15.478808 7638 tablet_copy_service.cc:342] P 0ddd486109f5434cace13d60d633ac04: Request end of tablet copy session 73a1f2ca1f544de9b0e3b37015f85328-d558bcaf20bd46548936d5fb946d7710 received from {username='slave'} at 127.29.82.68:47485
I20240502 21:28:15.478945 7638 tablet_copy_service.cc:433] P 0ddd486109f5434cace13d60d633ac04: ending tablet copy session 73a1f2ca1f544de9b0e3b37015f85328-d558bcaf20bd46548936d5fb946d7710 on tablet d558bcaf20bd46548936d5fb946d7710 with peer 73a1f2ca1f544de9b0e3b37015f85328
I20240502 21:28:15.555447 7348 raft_consensus.cc:1203] T a507a60fb2094bd3b9dc24bec06db867 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:15.600620 7617 raft_consensus.cc:1203] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:28:15.719138 4575 raft_consensus.cc:1203] T f800bfd495a94f2a952bc57a348304d0 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:15.721210 15578 ts_tablet_manager.cc:902] T 61e18d2b13644f6a99225c106c594914 P 71e1375b799b439fa3c83ca763c039c1: Initiating tablet copy from peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:28:15.748670 4839 raft_consensus.cc:1203] T d558bcaf20bd46548936d5fb946d7710 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:15.775406 15584 ts_tablet_manager.cc:902] T 2ae670e3d8ae49f2b8c7df65b45b179e P 398c3c8f3195492c900439ad5e1931bb: Initiating tablet copy from peer 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:28:15.784036 15578 tablet_copy_client.cc:308] T 61e18d2b13644f6a99225c106c594914 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.69:33193
I20240502 21:28:15.785599 7236 tablet_copy_service.cc:140] P 39981a8dfe94462b9a73ba489af4c714: Received BeginTabletCopySession request for tablet 61e18d2b13644f6a99225c106c594914 from peer 71e1375b799b439fa3c83ca763c039c1 ({username='slave'} at 127.29.82.65:42457)
I20240502 21:28:15.785773 7236 tablet_copy_service.cc:161] P 39981a8dfe94462b9a73ba489af4c714: Beginning new tablet copy session on tablet 61e18d2b13644f6a99225c106c594914 from peer 71e1375b799b439fa3c83ca763c039c1 at {username='slave'} at 127.29.82.65:42457: session id = 71e1375b799b439fa3c83ca763c039c1-61e18d2b13644f6a99225c106c594914
I20240502 21:28:15.788928 7236 tablet_copy_source_session.cc:205] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:15.791882 15578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61e18d2b13644f6a99225c106c594914. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:15.807685 15578 tablet_copy_client.cc:783] T 61e18d2b13644f6a99225c106c594914 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:15.808030 15578 tablet_copy_client.cc:647] T 61e18d2b13644f6a99225c106c594914 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:15.815182 15578 tablet_copy_client.cc:523] T 61e18d2b13644f6a99225c106c594914 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:15.821729 15578 tablet_bootstrap.cc:492] T 61e18d2b13644f6a99225c106c594914 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:28:15.848228 15578 tablet_bootstrap.cc:492] T 61e18d2b13644f6a99225c106c594914 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:15.848630 15578 tablet_bootstrap.cc:492] T 61e18d2b13644f6a99225c106c594914 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap complete.
I20240502 21:28:15.848887 15578 ts_tablet_manager.cc:1367] T 61e18d2b13644f6a99225c106c594914 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.002s
I20240502 21:28:15.849640 15578 raft_consensus.cc:348] T 61e18d2b13644f6a99225c106c594914 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:28:15.849861 15578 raft_consensus.cc:729] T 61e18d2b13644f6a99225c106c594914 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:28:15.850082 15578 consensus_queue.cc:260] T 61e18d2b13644f6a99225c106c594914 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:28:15.852119 15578 ts_tablet_manager.cc:1398] T 61e18d2b13644f6a99225c106c594914 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:28:15.853715 7236 tablet_copy_service.cc:342] P 39981a8dfe94462b9a73ba489af4c714: Request end of tablet copy session 71e1375b799b439fa3c83ca763c039c1-61e18d2b13644f6a99225c106c594914 received from {username='slave'} at 127.29.82.65:42457
I20240502 21:28:15.853852 7236 tablet_copy_service.cc:433] P 39981a8dfe94462b9a73ba489af4c714: ending tablet copy session 71e1375b799b439fa3c83ca763c039c1-61e18d2b13644f6a99225c106c594914 on tablet 61e18d2b13644f6a99225c106c594914 with peer 71e1375b799b439fa3c83ca763c039c1
I20240502 21:28:15.872154 15591 ts_tablet_manager.cc:902] T 93e0911f163f4629a4df0249c5eb8860 P 7a736a8ebb164ce0a051c0fccd76c3b8: Initiating tablet copy from peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:28:15.959964 15591 tablet_copy_client.cc:308] T 93e0911f163f4629a4df0249c5eb8860 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.68:42449
I20240502 21:28:15.960737 4859 tablet_copy_service.cc:140] P 73a1f2ca1f544de9b0e3b37015f85328: Received BeginTabletCopySession request for tablet 93e0911f163f4629a4df0249c5eb8860 from peer 7a736a8ebb164ce0a051c0fccd76c3b8 ({username='slave'} at 127.29.82.66:35827)
I20240502 21:28:15.960927 4859 tablet_copy_service.cc:161] P 73a1f2ca1f544de9b0e3b37015f85328: Beginning new tablet copy session on tablet 93e0911f163f4629a4df0249c5eb8860 from peer 7a736a8ebb164ce0a051c0fccd76c3b8 at {username='slave'} at 127.29.82.66:35827: session id = 7a736a8ebb164ce0a051c0fccd76c3b8-93e0911f163f4629a4df0249c5eb8860
I20240502 21:28:15.964454 4859 tablet_copy_source_session.cc:205] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:15.966487 15591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93e0911f163f4629a4df0249c5eb8860. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:15.977085 15591 tablet_copy_client.cc:783] T 93e0911f163f4629a4df0249c5eb8860 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:15.977434 15591 tablet_copy_client.cc:647] T 93e0911f163f4629a4df0249c5eb8860 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:15.980692 15591 tablet_copy_client.cc:523] T 93e0911f163f4629a4df0249c5eb8860 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:15.986225 15591 tablet_bootstrap.cc:492] T 93e0911f163f4629a4df0249c5eb8860 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:28:15.990971 15584 tablet_copy_client.cc:308] T 2ae670e3d8ae49f2b8c7df65b45b179e P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.71:33913
I20240502 21:28:16.004779 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:16.004884 7505 tablet_copy_service.cc:140] P 31a72286397c4edc9a63eca3412be168: Received BeginTabletCopySession request for tablet 2ae670e3d8ae49f2b8c7df65b45b179e from peer 398c3c8f3195492c900439ad5e1931bb ({username='slave'} at 127.29.82.67:43117)
I20240502 21:28:16.005046 7505 tablet_copy_service.cc:161] P 31a72286397c4edc9a63eca3412be168: Beginning new tablet copy session on tablet 2ae670e3d8ae49f2b8c7df65b45b179e from peer 398c3c8f3195492c900439ad5e1931bb at {username='slave'} at 127.29.82.67:43117: session id = 398c3c8f3195492c900439ad5e1931bb-2ae670e3d8ae49f2b8c7df65b45b179e
I20240502 21:28:16.010061 7505 tablet_copy_source_session.cc:205] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:16.014333 15584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ae670e3d8ae49f2b8c7df65b45b179e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:16.035818 15591 tablet_bootstrap.cc:492] T 93e0911f163f4629a4df0249c5eb8860 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:16.036406 15591 tablet_bootstrap.cc:492] T 93e0911f163f4629a4df0249c5eb8860 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap complete.
I20240502 21:28:16.036715 15591 ts_tablet_manager.cc:1367] T 93e0911f163f4629a4df0249c5eb8860 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.051s user 0.012s sys 0.004s
I20240502 21:28:16.040733 15591 raft_consensus.cc:348] T 93e0911f163f4629a4df0249c5eb8860 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:28:16.041009 15591 raft_consensus.cc:729] T 93e0911f163f4629a4df0249c5eb8860 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:28:16.042804 15591 consensus_queue.cc:260] T 93e0911f163f4629a4df0249c5eb8860 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:28:16.045223 15591 ts_tablet_manager.cc:1398] T 93e0911f163f4629a4df0249c5eb8860 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240502 21:28:16.046008 4859 tablet_copy_service.cc:342] P 73a1f2ca1f544de9b0e3b37015f85328: Request end of tablet copy session 7a736a8ebb164ce0a051c0fccd76c3b8-93e0911f163f4629a4df0249c5eb8860 received from {username='slave'} at 127.29.82.66:35827
I20240502 21:28:16.046133 4859 tablet_copy_service.cc:433] P 73a1f2ca1f544de9b0e3b37015f85328: ending tablet copy session 7a736a8ebb164ce0a051c0fccd76c3b8-93e0911f163f4629a4df0249c5eb8860 on tablet 93e0911f163f4629a4df0249c5eb8860 with peer 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:28:16.104168 7462 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 21:28:16.106158 4773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.67:34537, user_credentials={real_user=slave}} blocked reactor thread for 47394.4us
I20240502 21:28:16.115056 7194 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 21:28:16.119493 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:16.120256 15584 tablet_copy_client.cc:783] T 2ae670e3d8ae49f2b8c7df65b45b179e P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:16.120796 15584 tablet_copy_client.cc:647] T 2ae670e3d8ae49f2b8c7df65b45b179e P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:16.130105 4686 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:16.154754 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:16.162048 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:16.177332 15584 tablet_copy_client.cc:523] T 2ae670e3d8ae49f2b8c7df65b45b179e P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:16.185302 4817 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 21:28:16.185585 15584 tablet_bootstrap.cc:492] T 2ae670e3d8ae49f2b8c7df65b45b179e P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:28:16.223886 15584 tablet_bootstrap.cc:492] T 2ae670e3d8ae49f2b8c7df65b45b179e P 398c3c8f3195492c900439ad5e1931bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:16.224375 15584 tablet_bootstrap.cc:492] T 2ae670e3d8ae49f2b8c7df65b45b179e P 398c3c8f3195492c900439ad5e1931bb: Bootstrap complete.
I20240502 21:28:16.224691 15584 ts_tablet_manager.cc:1367] T 2ae670e3d8ae49f2b8c7df65b45b179e P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.002s
I20240502 21:28:16.225409 15584 raft_consensus.cc:348] T 2ae670e3d8ae49f2b8c7df65b45b179e P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:16.225739 15584 raft_consensus.cc:729] T 2ae670e3d8ae49f2b8c7df65b45b179e P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:28:16.226190 15584 consensus_queue.cc:260] T 2ae670e3d8ae49f2b8c7df65b45b179e P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:16.226797 15584 ts_tablet_manager.cc:1398] T 2ae670e3d8ae49f2b8c7df65b45b179e P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:28:16.230857 7505 tablet_copy_service.cc:342] P 31a72286397c4edc9a63eca3412be168: Request end of tablet copy session 398c3c8f3195492c900439ad5e1931bb-2ae670e3d8ae49f2b8c7df65b45b179e received from {username='slave'} at 127.29.82.67:43117
I20240502 21:28:16.231005 7505 tablet_copy_service.cc:433] P 31a72286397c4edc9a63eca3412be168: ending tablet copy session 398c3c8f3195492c900439ad5e1931bb-2ae670e3d8ae49f2b8c7df65b45b179e on tablet 2ae670e3d8ae49f2b8c7df65b45b179e with peer 398c3c8f3195492c900439ad5e1931bb
I20240502 21:28:16.285178 4442 raft_consensus.cc:1203] T 61e18d2b13644f6a99225c106c594914 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:16.295393 4703 raft_consensus.cc:1203] T 2ae670e3d8ae49f2b8c7df65b45b179e P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:16.321792 4575 raft_consensus.cc:1203] T 93e0911f163f4629a4df0249c5eb8860 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:16.687866 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:16.730060 7194 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 21:28:16.741676 7462 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 21:28:16.751260 4686 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:16.804092 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:16.812036 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:16.837806 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:16.867331 4817 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 21:28:17.058913 7214 consensus_queue.cc:237] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:28:17.061647 7348 raft_consensus.cc:1261] T 61e18d2b13644f6a99225c106c594914 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:17.061638 4443 raft_consensus.cc:1261] T 61e18d2b13644f6a99225c106c594914 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:17.062234 14846 consensus_queue.cc:1035] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:17.062522 15262 consensus_queue.cc:1035] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:17.069604 14846 raft_consensus.cc:2932] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:17.070231 7348 raft_consensus.cc:2932] T 61e18d2b13644f6a99225c106c594914 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:17.080327 4293 catalog_manager.cc:5484] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:17.087728 4443 raft_consensus.cc:2932] T 61e18d2b13644f6a99225c106c594914 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:17.100463 7597 tablet_service.cc:1514] Processing DeleteTablet for tablet 61e18d2b13644f6a99225c106c594914 with delete_type TABLET_DATA_TOMBSTONED (TS 0ddd486109f5434cace13d60d633ac04 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44444
I20240502 21:28:17.101462 15735 tablet_replica.cc:330] T 61e18d2b13644f6a99225c106c594914 P 0ddd486109f5434cace13d60d633ac04: stopping tablet replica
I20240502 21:28:17.101769 15735 raft_consensus.cc:2229] T 61e18d2b13644f6a99225c106c594914 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:17.102094 15735 raft_consensus.cc:2258] T 61e18d2b13644f6a99225c106c594914 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:17.105892 15735 ts_tablet_manager.cc:1875] T 61e18d2b13644f6a99225c106c594914 P 0ddd486109f5434cace13d60d633ac04: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:17.110587 15735 ts_tablet_manager.cc:1888] T 61e18d2b13644f6a99225c106c594914 P 0ddd486109f5434cace13d60d633ac04: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:17.110700 15735 log.cc:1196] T 61e18d2b13644f6a99225c106c594914 P 0ddd486109f5434cace13d60d633ac04: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/wals/61e18d2b13644f6a99225c106c594914
I20240502 21:28:17.111513 4285 catalog_manager.cc:4830] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): tablet 61e18d2b13644f6a99225c106c594914 (table rebalancer_extra_table_30 [id=b87f68aedb34492aade9e648ad016b2c]) successfully deleted
I20240502 21:28:17.529422 7462 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 21:28:17.564260 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:17.616905 4686 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:17.636152 7194 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 21:28:17.671808 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:17.723409 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:17.741871 4817 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 21:28:17.750067 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:17.907820 7484 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2ae670e3d8ae49f2b8c7df65b45b179e"
dest_uuid: "31a72286397c4edc9a63eca3412be168"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56038
I20240502 21:28:17.908131 7484 raft_consensus.cc:595] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:28:17.956185 15808 raft_consensus.cc:979] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168: : Instructing follower 398c3c8f3195492c900439ad5e1931bb to start an election
I20240502 21:28:17.956375 15808 raft_consensus.cc:1067] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Signalling peer 398c3c8f3195492c900439ad5e1931bb to start an election
I20240502 21:28:17.957190 4703 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2ae670e3d8ae49f2b8c7df65b45b179e"
dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
from {username='slave'} at 127.29.82.71:44205
I20240502 21:28:17.957432 4703 raft_consensus.cc:482] T 2ae670e3d8ae49f2b8c7df65b45b179e P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:28:17.957540 4703 raft_consensus.cc:3037] T 2ae670e3d8ae49f2b8c7df65b45b179e P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:17.960487 4703 raft_consensus.cc:504] T 2ae670e3d8ae49f2b8c7df65b45b179e P 398c3c8f3195492c900439ad5e1931bb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:17.962437 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae670e3d8ae49f2b8c7df65b45b179e" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:28:17.962698 7484 raft_consensus.cc:3032] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:28:17.962790 7484 raft_consensus.cc:729] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Running, Role: LEADER
I20240502 21:28:17.963133 7484 consensus_queue.cc:260] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:17.963393 7617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae670e3d8ae49f2b8c7df65b45b179e" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:28:17.963567 7617 raft_consensus.cc:3037] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:17.963595 7484 raft_consensus.cc:3037] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:17.965569 7617 raft_consensus.cc:2454] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 2.
I20240502 21:28:17.966323 7484 raft_consensus.cc:2454] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 2.
I20240502 21:28:17.967706 4639 leader_election.cc:302] T 2ae670e3d8ae49f2b8c7df65b45b179e P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168, 398c3c8f3195492c900439ad5e1931bb; no voters:
I20240502 21:28:17.968034 15538 raft_consensus.cc:2781] T 2ae670e3d8ae49f2b8c7df65b45b179e P 398c3c8f3195492c900439ad5e1931bb [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:28:17.967936 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae670e3d8ae49f2b8c7df65b45b179e" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:28:17.968161 7350 raft_consensus.cc:3037] T 2ae670e3d8ae49f2b8c7df65b45b179e P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:17.968183 15538 raft_consensus.cc:686] T 2ae670e3d8ae49f2b8c7df65b45b179e P 398c3c8f3195492c900439ad5e1931bb [term 2 LEADER]: Becoming Leader. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Running, Role: LEADER
I20240502 21:28:17.968473 15538 consensus_queue.cc:237] T 2ae670e3d8ae49f2b8c7df65b45b179e P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:17.970893 7350 raft_consensus.cc:2454] T 2ae670e3d8ae49f2b8c7df65b45b179e P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 2.
I20240502 21:28:17.972195 4703 leader_election.cc:288] T 2ae670e3d8ae49f2b8c7df65b45b179e P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 2 election: Requested vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:28:17.973639 4293 catalog_manager.cc:5484] T 2ae670e3d8ae49f2b8c7df65b45b179e P 398c3c8f3195492c900439ad5e1931bb reported cstate change: term changed from 1 to 2, leader changed from 31a72286397c4edc9a63eca3412be168 (127.29.82.71) to 398c3c8f3195492c900439ad5e1931bb (127.29.82.67). New cstate: current_term: 2 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:18.233814 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:18.267894 7194 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 21:28:18.305680 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:18.325187 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:18.354792 4817 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 21:28:18.359200 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:18.360451 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:18.364711 7350 raft_consensus.cc:1261] T 2ae670e3d8ae49f2b8c7df65b45b179e P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 21:28:18.369406 15538 consensus_queue.cc:1035] T 2ae670e3d8ae49f2b8c7df65b45b179e P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:18.386436 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:18.539701 7618 raft_consensus.cc:1261] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 21:28:18.540596 15538 consensus_queue.cc:1035] T 2ae670e3d8ae49f2b8c7df65b45b179e P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:18.555321 7484 raft_consensus.cc:1261] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 21:28:18.556845 15538 consensus_queue.cc:1035] T 2ae670e3d8ae49f2b8c7df65b45b179e P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:19.107266 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:19.191200 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:19.209873 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:19.239192 4817 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 21:28:19.251181 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:19.258148 7194 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 21:28:19.294682 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:19.462524 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:19.580818 7483 consensus_queue.cc:237] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:28:19.583758 4574 raft_consensus.cc:1261] T f800bfd495a94f2a952bc57a348304d0 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:19.583772 7617 raft_consensus.cc:1261] T f800bfd495a94f2a952bc57a348304d0 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:19.584247 15624 consensus_queue.cc:1035] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:19.584509 15409 consensus_queue.cc:1035] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:19.587185 15624 raft_consensus.cc:2932] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:19.590325 7617 raft_consensus.cc:2932] T f800bfd495a94f2a952bc57a348304d0 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:19.594002 4300 catalog_manager.cc:5484] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 reported cstate change: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:19.594460 4574 raft_consensus.cc:2932] T f800bfd495a94f2a952bc57a348304d0 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:19.613670 7194 tablet_service.cc:1514] Processing DeleteTablet for tablet f800bfd495a94f2a952bc57a348304d0 with delete_type TABLET_DATA_TOMBSTONED (TS 39981a8dfe94462b9a73ba489af4c714 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44652
I20240502 21:28:19.618340 15998 tablet_replica.cc:330] T f800bfd495a94f2a952bc57a348304d0 P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:28:19.618700 15998 raft_consensus.cc:2229] T f800bfd495a94f2a952bc57a348304d0 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:19.618952 15998 raft_consensus.cc:2258] T f800bfd495a94f2a952bc57a348304d0 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:19.619676 15998 ts_tablet_manager.cc:1875] T f800bfd495a94f2a952bc57a348304d0 P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:19.629024 15998 ts_tablet_manager.cc:1888] T f800bfd495a94f2a952bc57a348304d0 P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:19.629169 15998 log.cc:1196] T f800bfd495a94f2a952bc57a348304d0 P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/f800bfd495a94f2a952bc57a348304d0
I20240502 21:28:19.630451 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet f800bfd495a94f2a952bc57a348304d0 (table rebalancer_extra_table_2 [id=1b458a4121624eaaaf47feadd84aab8c]) successfully deleted
I20240502 21:28:19.920234 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:19.987478 7194 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 21:28:19.988782 4639 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.68:42449, user_credentials={real_user=slave}} blocked reactor thread for 92297.5us
I20240502 21:28:20.000054 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 21:28:20.022454 7551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.69:33193, user_credentials={real_user=slave}} blocked reactor thread for 52759us
I20240502 21:28:20.029641 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:20.033150 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:20.060685 4817 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 21:28:20.090983 7282 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.72:41223, user_credentials={real_user=slave}} blocked reactor thread for 65331.8us
I20240502 21:28:20.124730 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:20.135833 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:20.326207 7212 consensus_queue.cc:237] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:28:20.328234 7348 raft_consensus.cc:1261] T a507a60fb2094bd3b9dc24bec06db867 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:20.328246 4839 raft_consensus.cc:1261] T a507a60fb2094bd3b9dc24bec06db867 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:20.328963 15969 consensus_queue.cc:1035] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:20.329196 14846 consensus_queue.cc:1035] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:20.334444 16084 raft_consensus.cc:2932] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:20.337532 7348 raft_consensus.cc:2932] T a507a60fb2094bd3b9dc24bec06db867 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:20.341071 4294 catalog_manager.cc:5484] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:20.369730 4839 raft_consensus.cc:2932] T a507a60fb2094bd3b9dc24bec06db867 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:20.392724 7462 tablet_service.cc:1514] Processing DeleteTablet for tablet a507a60fb2094bd3b9dc24bec06db867 with delete_type TABLET_DATA_TOMBSTONED (TS 31a72286397c4edc9a63eca3412be168 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:37182
I20240502 21:28:20.404690 16095 tablet_replica.cc:330] T a507a60fb2094bd3b9dc24bec06db867 P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:28:20.404966 16095 raft_consensus.cc:2229] T a507a60fb2094bd3b9dc24bec06db867 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:20.405197 16095 raft_consensus.cc:2258] T a507a60fb2094bd3b9dc24bec06db867 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:20.406383 16095 ts_tablet_manager.cc:1875] T a507a60fb2094bd3b9dc24bec06db867 P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:20.415110 16095 ts_tablet_manager.cc:1888] T a507a60fb2094bd3b9dc24bec06db867 P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:20.415282 16095 log.cc:1196] T a507a60fb2094bd3b9dc24bec06db867 P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/a507a60fb2094bd3b9dc24bec06db867
I20240502 21:28:20.417842 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet a507a60fb2094bd3b9dc24bec06db867 (table rebalancer_extra_table_34 [id=e1c7649d4d1248839d2699012aea3ee5]) successfully deleted
I20240502 21:28:20.705981 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:20.742359 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:20.749193 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:20.752941 4817 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 21:28:20.769953 7194 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 21:28:20.836530 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:20.863186 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:20.911085 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:20.961781 4289 maintenance_manager.cc:419] P e4b7d987f0ff4a3589447bd47c1ddd44: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033382
I20240502 21:28:21.110577 4288 maintenance_manager.cc:643] P e4b7d987f0ff4a3589447bd47c1ddd44: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.148s user 0.079s sys 0.041s Metrics: {"bytes_written":103360,"cfile_init":1,"dirs.queue_time_us":37540,"dirs.run_cpu_time_us":285,"dirs.run_wall_time_us":938,"drs_written":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1474,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":287,"thread_start_us":37401,"threads_started":1}
I20240502 21:28:21.112154 4289 maintenance_manager.cc:419] P e4b7d987f0ff4a3589447bd47c1ddd44: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 32045 bytes on disk
I20240502 21:28:21.112993 4288 maintenance_manager.cc:643] P e4b7d987f0ff4a3589447bd47c1ddd44: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240502 21:28:21.141420 7618 consensus_queue.cc:237] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:28:21.143294 7212 raft_consensus.cc:1261] T cfceb76b56924833bd7e6b3901573ee7 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:21.143280 4838 raft_consensus.cc:1261] T cfceb76b56924833bd7e6b3901573ee7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:21.143836 16060 consensus_queue.cc:1035] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:21.144438 16060 consensus_queue.cc:1035] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:21.147111 16060 raft_consensus.cc:2932] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:21.147648 4838 raft_consensus.cc:2932] T cfceb76b56924833bd7e6b3901573ee7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:21.151297 7212 raft_consensus.cc:2932] T cfceb76b56924833bd7e6b3901573ee7 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:21.153708 4296 catalog_manager.cc:5484] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: config changed from index 2 to 3, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:21.168540 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet cfceb76b56924833bd7e6b3901573ee7 with delete_type TABLET_DATA_TOMBSTONED (TS 398c3c8f3195492c900439ad5e1931bb not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44764
I20240502 21:28:21.169127 16188 tablet_replica.cc:330] T cfceb76b56924833bd7e6b3901573ee7 P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:28:21.169405 16188 raft_consensus.cc:2229] T cfceb76b56924833bd7e6b3901573ee7 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:21.169798 16188 raft_consensus.cc:2258] T cfceb76b56924833bd7e6b3901573ee7 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:21.170641 16188 ts_tablet_manager.cc:1875] T cfceb76b56924833bd7e6b3901573ee7 P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:21.178259 16188 ts_tablet_manager.cc:1888] T cfceb76b56924833bd7e6b3901573ee7 P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:21.178447 16188 log.cc:1196] T cfceb76b56924833bd7e6b3901573ee7 P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/cfceb76b56924833bd7e6b3901573ee7
I20240502 21:28:21.179154 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet cfceb76b56924833bd7e6b3901573ee7 (table rebalancer_extra_table_52 [id=93af5a5216354f909ee81e7cc68d1068]) successfully deleted
I20240502 21:28:21.584439 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:21.589310 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:21.614807 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 21:28:21.709546 4640 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.72:41223, user_credentials={real_user=slave}} blocked reactor thread for 77984.3us
I20240502 21:28:21.713507 4817 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 21:28:21.741770 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:21.763680 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:21.774489 7194 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 21:28:21.847748 4378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.67:34537, user_credentials={real_user=slave}} blocked reactor thread for 93711.2us
I20240502 21:28:21.876456 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:22.101657 7618 consensus_queue.cc:237] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:28:22.106124 7484 raft_consensus.cc:1261] T d558bcaf20bd46548936d5fb946d7710 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:22.106278 4839 raft_consensus.cc:1261] T d558bcaf20bd46548936d5fb946d7710 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:22.107080 16060 consensus_queue.cc:1035] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:22.107681 16060 consensus_queue.cc:1035] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:22.119052 16060 raft_consensus.cc:2932] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:22.121289 7483 raft_consensus.cc:2932] T d558bcaf20bd46548936d5fb946d7710 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:22.138855 4296 catalog_manager.cc:5484] T d558bcaf20bd46548936d5fb946d7710 P 31a72286397c4edc9a63eca3412be168 reported cstate change: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:22.144862 4839 raft_consensus.cc:2932] T d558bcaf20bd46548936d5fb946d7710 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:22.159376 7328 tablet_service.cc:1514] Processing DeleteTablet for tablet d558bcaf20bd46548936d5fb946d7710 with delete_type TABLET_DATA_TOMBSTONED (TS d0456c5e55a542128688904e003d403c not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47078
I20240502 21:28:22.160177 16293 tablet_replica.cc:330] T d558bcaf20bd46548936d5fb946d7710 P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:28:22.160580 16293 raft_consensus.cc:2229] T d558bcaf20bd46548936d5fb946d7710 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:22.160903 16293 raft_consensus.cc:2258] T d558bcaf20bd46548936d5fb946d7710 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:22.161675 16293 ts_tablet_manager.cc:1875] T d558bcaf20bd46548936d5fb946d7710 P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:22.172489 16293 ts_tablet_manager.cc:1888] T d558bcaf20bd46548936d5fb946d7710 P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:22.172681 16293 log.cc:1196] T d558bcaf20bd46548936d5fb946d7710 P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/d558bcaf20bd46548936d5fb946d7710
I20240502 21:28:22.174425 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet d558bcaf20bd46548936d5fb946d7710 (table rebalancer_dynamic_table [id=52a205feacb541d3a82d26d91459b07c]) successfully deleted
I20240502 21:28:22.566895 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:22.639029 7194 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 21:28:22.664450 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:22.671813 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:22.680238 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:22.689121 4817 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 21:28:22.691665 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:22.713866 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:22.919368 4838 consensus_queue.cc:237] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:28:22.921667 7616 raft_consensus.cc:1261] T 104c676bff964b8b8566b2200ce238c7 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:22.921846 4705 raft_consensus.cc:1261] T 104c676bff964b8b8566b2200ce238c7 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:22.922204 15322 consensus_queue.cc:1035] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:22.922466 16266 consensus_queue.cc:1035] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:22.925418 15322 raft_consensus.cc:2932] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:22.932489 4705 raft_consensus.cc:2932] T 104c676bff964b8b8566b2200ce238c7 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:22.938846 4296 catalog_manager.cc:5484] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:22.955392 7618 raft_consensus.cc:2932] T 104c676bff964b8b8566b2200ce238c7 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:22.959986 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet 104c676bff964b8b8566b2200ce238c7 with delete_type TABLET_DATA_TOMBSTONED (TS 7a736a8ebb164ce0a051c0fccd76c3b8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60018
I20240502 21:28:22.962981 16388 tablet_replica.cc:330] T 104c676bff964b8b8566b2200ce238c7 P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:28:22.963270 16388 raft_consensus.cc:2229] T 104c676bff964b8b8566b2200ce238c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:22.963671 16388 raft_consensus.cc:2258] T 104c676bff964b8b8566b2200ce238c7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:22.964551 16388 ts_tablet_manager.cc:1875] T 104c676bff964b8b8566b2200ce238c7 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:22.975522 16388 ts_tablet_manager.cc:1888] T 104c676bff964b8b8566b2200ce238c7 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:28:22.975802 16388 log.cc:1196] T 104c676bff964b8b8566b2200ce238c7 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/104c676bff964b8b8566b2200ce238c7
I20240502 21:28:22.976547 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet 104c676bff964b8b8566b2200ce238c7 (table rebalance_test_table_17 [id=cc837d2089e24e0fa0a7c398ca3eeca8]) successfully deleted
I20240502 21:28:23.281682 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:23.283528 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:23.301071 7194 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 21:28:23.307642 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:23.400244 4817 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 21:28:23.463160 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:23.481220 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:23.517879 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:23.672793 7214 consensus_queue.cc:237] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:28:23.675164 7484 raft_consensus.cc:1261] T be423032bc4b459b8e8e63a2379e37ac P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:23.675422 7350 raft_consensus.cc:1261] T be423032bc4b459b8e8e63a2379e37ac P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:23.676155 16448 consensus_queue.cc:1035] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:23.676483 16355 consensus_queue.cc:1035] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:23.681211 16448 raft_consensus.cc:2932] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:23.681613 7483 raft_consensus.cc:2932] T be423032bc4b459b8e8e63a2379e37ac P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:23.682224 7350 raft_consensus.cc:2932] T be423032bc4b459b8e8e63a2379e37ac P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:23.687722 4296 catalog_manager.cc:5484] T be423032bc4b459b8e8e63a2379e37ac P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:23.695853 7597 tablet_service.cc:1514] Processing DeleteTablet for tablet be423032bc4b459b8e8e63a2379e37ac with delete_type TABLET_DATA_TOMBSTONED (TS 0ddd486109f5434cace13d60d633ac04 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44444
I20240502 21:28:23.708014 16483 tablet_replica.cc:330] T be423032bc4b459b8e8e63a2379e37ac P 0ddd486109f5434cace13d60d633ac04: stopping tablet replica
I20240502 21:28:23.708923 16483 raft_consensus.cc:2229] T be423032bc4b459b8e8e63a2379e37ac P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:23.711452 16483 raft_consensus.cc:2258] T be423032bc4b459b8e8e63a2379e37ac P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:23.712402 16483 ts_tablet_manager.cc:1875] T be423032bc4b459b8e8e63a2379e37ac P 0ddd486109f5434cace13d60d633ac04: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:23.720594 16483 ts_tablet_manager.cc:1888] T be423032bc4b459b8e8e63a2379e37ac P 0ddd486109f5434cace13d60d633ac04: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:23.720858 16483 log.cc:1196] T be423032bc4b459b8e8e63a2379e37ac P 0ddd486109f5434cace13d60d633ac04: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/wals/be423032bc4b459b8e8e63a2379e37ac
I20240502 21:28:23.721633 4285 catalog_manager.cc:4830] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): tablet be423032bc4b459b8e8e63a2379e37ac (table rebalancer_extra_table_38 [id=569b30a95e114b50ada524f1380f198e]) successfully deleted
I20240502 21:28:24.101752 7194 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 21:28:24.105579 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:24.123975 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:24.163390 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:24.180735 4817 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 21:28:24.201694 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:24.236687 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:24.244824 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:24.446293 4703 consensus_queue.cc:237] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:28:24.448638 7483 raft_consensus.cc:1261] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:24.448655 4439 raft_consensus.cc:1261] T fd610e340cd3409397d8f23b623b4074 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:24.449177 16436 consensus_queue.cc:1035] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:24.449483 16261 consensus_queue.cc:1035] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:24.452368 16261 raft_consensus.cc:2932] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:24.453055 4439 raft_consensus.cc:2932] T fd610e340cd3409397d8f23b623b4074 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:24.455487 7483 raft_consensus.cc:2932] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:24.461580 4297 catalog_manager.cc:5484] T fd610e340cd3409397d8f23b623b4074 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: config changed from index 2 to 3, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:24.475975 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet fd610e340cd3409397d8f23b623b4074 with delete_type TABLET_DATA_TOMBSTONED (TS 7a736a8ebb164ce0a051c0fccd76c3b8 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:60018
I20240502 21:28:24.486984 16572 tablet_replica.cc:330] T fd610e340cd3409397d8f23b623b4074 P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:28:24.487258 16572 raft_consensus.cc:2229] T fd610e340cd3409397d8f23b623b4074 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:24.487475 16572 raft_consensus.cc:2258] T fd610e340cd3409397d8f23b623b4074 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:24.488220 16572 ts_tablet_manager.cc:1875] T fd610e340cd3409397d8f23b623b4074 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:24.494228 16572 ts_tablet_manager.cc:1888] T fd610e340cd3409397d8f23b623b4074 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:24.494354 16572 log.cc:1196] T fd610e340cd3409397d8f23b623b4074 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/fd610e340cd3409397d8f23b623b4074
I20240502 21:28:24.495096 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet fd610e340cd3409397d8f23b623b4074 (table rebalancer_extra_table_58 [id=9e3a4d2e56654ed784b01da7d60bdc59]) successfully deleted
I20240502 21:28:24.815380 7194 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 21:28:24.824688 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:24.829469 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:24.874291 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:24.906487 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:24.951959 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:24.952654 4817 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 21:28:25.011004 7419 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.69:33193, user_credentials={real_user=slave}} blocked reactor thread for 56480us
I20240502 21:28:25.022054 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240502 21:28:25.123800 4377 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.70:40719, user_credentials={real_user=slave}} blocked reactor thread for 91448us
I20240502 21:28:25.208370 4703 consensus_queue.cc:237] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:28:25.212707 4575 raft_consensus.cc:1261] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:25.212885 7618 raft_consensus.cc:1261] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:25.213256 16261 consensus_queue.cc:1035] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:25.213567 16436 consensus_queue.cc:1035] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:25.216589 16436 raft_consensus.cc:2932] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:25.216974 4575 raft_consensus.cc:2932] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:25.220008 7618 raft_consensus.cc:2932] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:25.225483 4294 catalog_manager.cc:5484] T 38ce97f9f9ac4398a16958931ef7262f P 398c3c8f3195492c900439ad5e1931bb reported cstate change: config changed from index 4 to 5, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) evicted. New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:25.236295 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet 38ce97f9f9ac4398a16958931ef7262f with delete_type TABLET_DATA_TOMBSTONED (TS 71e1375b799b439fa3c83ca763c039c1 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35984
I20240502 21:28:25.240073 16679 tablet_replica.cc:330] T 38ce97f9f9ac4398a16958931ef7262f P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:28:25.240365 16679 raft_consensus.cc:2229] T 38ce97f9f9ac4398a16958931ef7262f P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:25.240759 16679 raft_consensus.cc:2258] T 38ce97f9f9ac4398a16958931ef7262f P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:25.241763 16679 ts_tablet_manager.cc:1875] T 38ce97f9f9ac4398a16958931ef7262f P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:25.249707 16679 ts_tablet_manager.cc:1888] T 38ce97f9f9ac4398a16958931ef7262f P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:28:25.249838 16679 log.cc:1196] T 38ce97f9f9ac4398a16958931ef7262f P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/38ce97f9f9ac4398a16958931ef7262f
I20240502 21:28:25.250528 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet 38ce97f9f9ac4398a16958931ef7262f (table rebalance_test_table_19 [id=879d91ac491a4b2dbed04af44dc5e8de]) successfully deleted
I20240502 21:28:25.661702 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:25.678947 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:25.685236 7194 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 21:28:25.748548 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:25.749897 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:25.790860 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:25.801887 4817 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 21:28:25.840265 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:25.972885 7214 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "831d24800da54da9b22f9e1bca474755"
dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54530
I20240502 21:28:25.973099 7214 raft_consensus.cc:595] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:28:26.069921 16448 raft_consensus.cc:979] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714: : Instructing follower 73a1f2ca1f544de9b0e3b37015f85328 to start an election
I20240502 21:28:26.070127 16448 raft_consensus.cc:1067] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Signalling peer 73a1f2ca1f544de9b0e3b37015f85328 to start an election
I20240502 21:28:26.072679 4838 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "831d24800da54da9b22f9e1bca474755"
dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
from {username='slave'} at 127.29.82.69:57807
I20240502 21:28:26.072880 4838 raft_consensus.cc:482] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:28:26.072962 4838 raft_consensus.cc:3037] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:26.076700 4838 raft_consensus.cc:504] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:28:26.079246 4703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831d24800da54da9b22f9e1bca474755" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
I20240502 21:28:26.079488 4703 raft_consensus.cc:3037] T 831d24800da54da9b22f9e1bca474755 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:26.082403 4703 raft_consensus.cc:2454] T 831d24800da54da9b22f9e1bca474755 P 398c3c8f3195492c900439ad5e1931bb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 2.
I20240502 21:28:26.082677 4838 leader_election.cc:288] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 2 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:28:26.083374 7214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831d24800da54da9b22f9e1bca474755" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:28:26.083611 7214 raft_consensus.cc:3032] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:28:26.083683 7214 raft_consensus.cc:729] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:28:26.083722 7348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831d24800da54da9b22f9e1bca474755" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:28:26.083909 7348 raft_consensus.cc:3037] T 831d24800da54da9b22f9e1bca474755 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:26.083925 7214 consensus_queue.cc:260] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:28:26.084477 7214 raft_consensus.cc:3037] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:26.086542 7348 raft_consensus.cc:2454] T 831d24800da54da9b22f9e1bca474755 P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 2.
I20240502 21:28:26.091046 4773 leader_election.cc:302] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 73a1f2ca1f544de9b0e3b37015f85328, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:28:26.091362 16377 raft_consensus.cc:2781] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:28:26.097573 7214 raft_consensus.cc:2454] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 2.
I20240502 21:28:26.105418 16377 raft_consensus.cc:686] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328 [term 2 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:28:26.105815 16377 consensus_queue.cc:237] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:28:26.129256 4293 catalog_manager.cc:5484] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 1 to 2, leader changed from 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 2 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:26.301285 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:26.339172 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:26.348100 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:26.366034 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:26.429337 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:26.497516 7348 raft_consensus.cc:1261] T 831d24800da54da9b22f9e1bca474755 P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 21:28:26.498734 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:26.499155 16786 consensus_queue.cc:1035] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:26.544009 7214 raft_consensus.cc:1261] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [term 2 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 21:28:26.544575 16377 consensus_queue.cc:1035] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:26.602795 4705 raft_consensus.cc:1261] T 831d24800da54da9b22f9e1bca474755 P 398c3c8f3195492c900439ad5e1931bb [term 2 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 21:28:26.606599 16377 consensus_queue.cc:1035] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:26.621714 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:26.660452 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:27.081013 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:27.141974 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:27.190299 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:27.227864 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:27.228250 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 21:28:27.248570 7549 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.65:42781, user_credentials={real_user=slave}} blocked reactor thread for 45067.1us
W20240502 21:28:27.257757 7150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.70:40719, user_credentials={real_user=slave}} blocked reactor thread for 92907.2us
I20240502 21:28:27.262753 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:27.267804 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:27.301075 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:27.464138 7212 consensus_queue.cc:237] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:27.466459 4573 raft_consensus.cc:1261] T 42edb2b3ff094c71872fde7949b61b3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:27.466540 4703 raft_consensus.cc:1261] T 42edb2b3ff094c71872fde7949b61b3b P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:27.467080 16897 consensus_queue.cc:1035] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:27.467398 16709 consensus_queue.cc:1035] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:27.472821 16709 raft_consensus.cc:2932] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:27.473861 4703 raft_consensus.cc:2932] T 42edb2b3ff094c71872fde7949b61b3b P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:27.474048 4574 raft_consensus.cc:2932] T 42edb2b3ff094c71872fde7949b61b3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:27.482956 4291 catalog_manager.cc:5484] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: config changed from index 2 to 3, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) evicted. New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:27.496846 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet 42edb2b3ff094c71872fde7949b61b3b with delete_type TABLET_DATA_TOMBSTONED (TS 71e1375b799b439fa3c83ca763c039c1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:35984
I20240502 21:28:27.499118 16929 tablet_replica.cc:330] T 42edb2b3ff094c71872fde7949b61b3b P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:28:27.499492 16929 raft_consensus.cc:2229] T 42edb2b3ff094c71872fde7949b61b3b P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:27.499786 16929 raft_consensus.cc:2258] T 42edb2b3ff094c71872fde7949b61b3b P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:27.501531 16929 ts_tablet_manager.cc:1875] T 42edb2b3ff094c71872fde7949b61b3b P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:27.509965 16929 ts_tablet_manager.cc:1888] T 42edb2b3ff094c71872fde7949b61b3b P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:27.510107 16929 log.cc:1196] T 42edb2b3ff094c71872fde7949b61b3b P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/42edb2b3ff094c71872fde7949b61b3b
I20240502 21:28:27.510977 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet 42edb2b3ff094c71872fde7949b61b3b (table rebalancer_extra_table_22 [id=50e84343189b44c5bd6158fbd7ba5394]) successfully deleted
I20240502 21:28:27.561200 7483 consensus_queue.cc:237] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:28:27.564010 4439 raft_consensus.cc:1261] T 20b714b400824ecf86890f92aeff19be P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:27.564277 4838 raft_consensus.cc:1261] T 20b714b400824ecf86890f92aeff19be P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:28:27.564399 7418 consensus_peers.cc:485] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 -> Peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): Couldn't send request to peer 0ddd486109f5434cace13d60d633ac04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 20b714b400824ecf86890f92aeff19be. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:27.564846 15409 consensus_queue.cc:1035] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:27.565130 16644 consensus_queue.cc:1035] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:27.583627 16644 raft_consensus.cc:2932] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:27.584497 4439 raft_consensus.cc:2932] T 20b714b400824ecf86890f92aeff19be P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:27.590118 4838 raft_consensus.cc:2932] T 20b714b400824ecf86890f92aeff19be P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:27.591271 4292 catalog_manager.cc:5484] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 reported cstate change: config changed from index -1 to 2, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:27.626539 4838 consensus_queue.cc:237] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:28:27.629604 4574 raft_consensus.cc:1261] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:28:27.629657 4770 consensus_peers.cc:485] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): Couldn't send request to peer 31a72286397c4edc9a63eca3412be168. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d0fcf18fd154a72a3bb2f0744febfc7. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:27.629633 4705 raft_consensus.cc:1261] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:28:27.630215 16904 consensus_queue.cc:1035] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:27.630491 16377 consensus_queue.cc:1035] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:27.633270 16377 raft_consensus.cc:2932] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:27.633878 4705 raft_consensus.cc:2932] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:27.637557 4574 raft_consensus.cc:2932] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:27.641386 4292 catalog_manager.cc:5484] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index -1 to 4, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:27.682200 4705 consensus_queue.cc:237] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:28:27.689254 7483 raft_consensus.cc:1261] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:27.689512 7349 raft_consensus.cc:1261] T 0f822150741f4ec1a3a9472572160359 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:27.689849 16668 consensus_queue.cc:1035] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:28:27.690157 4641 consensus_peers.cc:485] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb -> Peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Couldn't send request to peer 39981a8dfe94462b9a73ba489af4c714. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f822150741f4ec1a3a9472572160359. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:27.690493 16668 consensus_queue.cc:1035] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:27.694106 16821 raft_consensus.cc:2932] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:27.694566 7483 raft_consensus.cc:2932] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:27.701303 7349 raft_consensus.cc:2932] T 0f822150741f4ec1a3a9472572160359 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:27.701553 4292 catalog_manager.cc:5484] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 reported cstate change: config changed from index -1 to 2, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:27.746001 7212 consensus_queue.cc:237] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:27.748553 4440 raft_consensus.cc:1261] T 94e66998ede944a6b00478270029ebcf P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:27.748672 7349 raft_consensus.cc:1261] T 94e66998ede944a6b00478270029ebcf P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:28:27.748991 7151 consensus_peers.cc:485] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 94e66998ede944a6b00478270029ebcf. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:27.749240 16709 consensus_queue.cc:1035] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:27.749506 16897 consensus_queue.cc:1035] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:27.753228 16709 raft_consensus.cc:2932] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:27.753612 7349 raft_consensus.cc:2932] T 94e66998ede944a6b00478270029ebcf P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:27.757601 4440 raft_consensus.cc:2932] T 94e66998ede944a6b00478270029ebcf P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:27.760727 4292 catalog_manager.cc:5484] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:27.823350 4839 consensus_queue.cc:237] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:28:27.837165 4574 raft_consensus.cc:1261] T e95d3785435e450688d87a0ce02d0a34 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:28:27.838014 16786 consensus_queue.cc:1035] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:28:27.839031 4770 consensus_peers.cc:485] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): Couldn't send request to peer 31a72286397c4edc9a63eca3412be168. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e95d3785435e450688d87a0ce02d0a34. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:27.843761 4705 raft_consensus.cc:1261] T e95d3785435e450688d87a0ce02d0a34 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:28:27.850244 16786 consensus_queue.cc:1035] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:27.877223 16377 raft_consensus.cc:2932] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:27.877877 4705 raft_consensus.cc:2932] T e95d3785435e450688d87a0ce02d0a34 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:27.884652 4300 catalog_manager.cc:5484] T e95d3785435e450688d87a0ce02d0a34 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: config changed from index -1 to 4, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:27.884891 4574 raft_consensus.cc:2932] T e95d3785435e450688d87a0ce02d0a34 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:28.103721 17069 ts_tablet_manager.cc:902] T 20b714b400824ecf86890f92aeff19be P 0ddd486109f5434cace13d60d633ac04: Initiating tablet copy from peer 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:28:28.180248 17069 tablet_copy_client.cc:308] T 20b714b400824ecf86890f92aeff19be P 0ddd486109f5434cace13d60d633ac04: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.71:33913
I20240502 21:28:28.181929 7505 tablet_copy_service.cc:140] P 31a72286397c4edc9a63eca3412be168: Received BeginTabletCopySession request for tablet 20b714b400824ecf86890f92aeff19be from peer 0ddd486109f5434cace13d60d633ac04 ({username='slave'} at 127.29.82.72:45539)
I20240502 21:28:28.182087 7505 tablet_copy_service.cc:161] P 31a72286397c4edc9a63eca3412be168: Beginning new tablet copy session on tablet 20b714b400824ecf86890f92aeff19be from peer 0ddd486109f5434cace13d60d633ac04 at {username='slave'} at 127.29.82.72:45539: session id = 0ddd486109f5434cace13d60d633ac04-20b714b400824ecf86890f92aeff19be
I20240502 21:28:28.184422 7505 tablet_copy_source_session.cc:205] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:28.186342 17069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20b714b400824ecf86890f92aeff19be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:28.194561 17069 tablet_copy_client.cc:783] T 20b714b400824ecf86890f92aeff19be P 0ddd486109f5434cace13d60d633ac04: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:28.194905 17069 tablet_copy_client.cc:647] T 20b714b400824ecf86890f92aeff19be P 0ddd486109f5434cace13d60d633ac04: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:28.196703 17069 tablet_copy_client.cc:523] T 20b714b400824ecf86890f92aeff19be P 0ddd486109f5434cace13d60d633ac04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:28.202993 17069 tablet_bootstrap.cc:492] T 20b714b400824ecf86890f92aeff19be P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:28:28.219319 17095 ts_tablet_manager.cc:902] T 0f822150741f4ec1a3a9472572160359 P 39981a8dfe94462b9a73ba489af4c714: Initiating tablet copy from peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:28:28.224157 17069 tablet_bootstrap.cc:492] T 20b714b400824ecf86890f92aeff19be P 0ddd486109f5434cace13d60d633ac04: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:28.224588 17069 tablet_bootstrap.cc:492] T 20b714b400824ecf86890f92aeff19be P 0ddd486109f5434cace13d60d633ac04: Bootstrap complete.
I20240502 21:28:28.224860 17069 ts_tablet_manager.cc:1367] T 20b714b400824ecf86890f92aeff19be P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.000s
I20240502 21:28:28.225636 17069 raft_consensus.cc:348] T 20b714b400824ecf86890f92aeff19be P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:28:28.225924 17069 raft_consensus.cc:729] T 20b714b400824ecf86890f92aeff19be P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:28:28.226171 17069 consensus_queue.cc:260] T 20b714b400824ecf86890f92aeff19be P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:28:28.229313 17097 ts_tablet_manager.cc:902] T 94e66998ede944a6b00478270029ebcf P 398c3c8f3195492c900439ad5e1931bb: Initiating tablet copy from peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:28:28.232471 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:28.235869 17069 ts_tablet_manager.cc:1398] T 20b714b400824ecf86890f92aeff19be P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.011s user 0.002s sys 0.001s
I20240502 21:28:28.243726 7505 tablet_copy_service.cc:342] P 31a72286397c4edc9a63eca3412be168: Request end of tablet copy session 0ddd486109f5434cace13d60d633ac04-20b714b400824ecf86890f92aeff19be received from {username='slave'} at 127.29.82.72:45539
I20240502 21:28:28.243845 7505 tablet_copy_service.cc:433] P 31a72286397c4edc9a63eca3412be168: ending tablet copy session 0ddd486109f5434cace13d60d633ac04-20b714b400824ecf86890f92aeff19be on tablet 20b714b400824ecf86890f92aeff19be with peer 0ddd486109f5434cace13d60d633ac04
I20240502 21:28:28.243935 17102 ts_tablet_manager.cc:902] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 31a72286397c4edc9a63eca3412be168: Initiating tablet copy from peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:28:28.253973 17097 tablet_copy_client.cc:308] T 94e66998ede944a6b00478270029ebcf P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.69:33193
I20240502 21:28:28.260246 7236 tablet_copy_service.cc:140] P 39981a8dfe94462b9a73ba489af4c714: Received BeginTabletCopySession request for tablet 94e66998ede944a6b00478270029ebcf from peer 398c3c8f3195492c900439ad5e1931bb ({username='slave'} at 127.29.82.67:59759)
I20240502 21:28:28.260428 7236 tablet_copy_service.cc:161] P 39981a8dfe94462b9a73ba489af4c714: Beginning new tablet copy session on tablet 94e66998ede944a6b00478270029ebcf from peer 398c3c8f3195492c900439ad5e1931bb at {username='slave'} at 127.29.82.67:59759: session id = 398c3c8f3195492c900439ad5e1931bb-94e66998ede944a6b00478270029ebcf
I20240502 21:28:28.260443 17102 tablet_copy_client.cc:308] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 31a72286397c4edc9a63eca3412be168: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.68:42449
I20240502 21:28:28.261135 4859 tablet_copy_service.cc:140] P 73a1f2ca1f544de9b0e3b37015f85328: Received BeginTabletCopySession request for tablet 1d0fcf18fd154a72a3bb2f0744febfc7 from peer 31a72286397c4edc9a63eca3412be168 ({username='slave'} at 127.29.82.71:36305)
I20240502 21:28:28.261284 4859 tablet_copy_service.cc:161] P 73a1f2ca1f544de9b0e3b37015f85328: Beginning new tablet copy session on tablet 1d0fcf18fd154a72a3bb2f0744febfc7 from peer 31a72286397c4edc9a63eca3412be168 at {username='slave'} at 127.29.82.71:36305: session id = 31a72286397c4edc9a63eca3412be168-1d0fcf18fd154a72a3bb2f0744febfc7
I20240502 21:28:28.263397 7236 tablet_copy_source_session.cc:205] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:28.263988 4859 tablet_copy_source_session.cc:205] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:28.266336 17097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e66998ede944a6b00478270029ebcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:28.284209 17102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d0fcf18fd154a72a3bb2f0744febfc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:28.295982 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:28.296078 17095 tablet_copy_client.cc:308] T 0f822150741f4ec1a3a9472572160359 P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.67:34537
I20240502 21:28:28.302493 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:28.310988 4727 tablet_copy_service.cc:140] P 398c3c8f3195492c900439ad5e1931bb: Received BeginTabletCopySession request for tablet 0f822150741f4ec1a3a9472572160359 from peer 39981a8dfe94462b9a73ba489af4c714 ({username='slave'} at 127.29.82.69:46901)
I20240502 21:28:28.311154 4727 tablet_copy_service.cc:161] P 398c3c8f3195492c900439ad5e1931bb: Beginning new tablet copy session on tablet 0f822150741f4ec1a3a9472572160359 from peer 39981a8dfe94462b9a73ba489af4c714 at {username='slave'} at 127.29.82.69:46901: session id = 39981a8dfe94462b9a73ba489af4c714-0f822150741f4ec1a3a9472572160359
I20240502 21:28:28.313935 4727 tablet_copy_source_session.cc:205] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:28.316541 17095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f822150741f4ec1a3a9472572160359. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:28.320060 17097 tablet_copy_client.cc:783] T 94e66998ede944a6b00478270029ebcf P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:28.320307 17097 tablet_copy_client.cc:647] T 94e66998ede944a6b00478270029ebcf P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:28.322764 17102 tablet_copy_client.cc:783] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 31a72286397c4edc9a63eca3412be168: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:28.323021 17102 tablet_copy_client.cc:647] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 31a72286397c4edc9a63eca3412be168: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:28.337432 17095 tablet_copy_client.cc:783] T 0f822150741f4ec1a3a9472572160359 P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:28.337729 17095 tablet_copy_client.cc:647] T 0f822150741f4ec1a3a9472572160359 P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:28.346760 17102 tablet_copy_client.cc:523] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 31a72286397c4edc9a63eca3412be168: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:28.350422 17097 tablet_copy_client.cc:523] T 94e66998ede944a6b00478270029ebcf P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:28.351401 17102 tablet_bootstrap.cc:492] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:28:28.354380 17097 tablet_bootstrap.cc:492] T 94e66998ede944a6b00478270029ebcf P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:28:28.368638 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:28.369635 17095 tablet_copy_client.cc:523] T 0f822150741f4ec1a3a9472572160359 P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:28.376231 17095 tablet_bootstrap.cc:492] T 0f822150741f4ec1a3a9472572160359 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:28:28.390671 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:28.411512 17124 ts_tablet_manager.cc:902] T e95d3785435e450688d87a0ce02d0a34 P 31a72286397c4edc9a63eca3412be168: Initiating tablet copy from peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:28:28.437104 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:28.450381 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:28.457195 17095 tablet_bootstrap.cc:492] T 0f822150741f4ec1a3a9472572160359 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:28.466517 17095 tablet_bootstrap.cc:492] T 0f822150741f4ec1a3a9472572160359 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap complete.
I20240502 21:28:28.466876 17095 ts_tablet_manager.cc:1367] T 0f822150741f4ec1a3a9472572160359 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.091s user 0.008s sys 0.003s
I20240502 21:28:28.467854 17095 raft_consensus.cc:348] T 0f822150741f4ec1a3a9472572160359 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:28:28.468178 17095 raft_consensus.cc:729] T 0f822150741f4ec1a3a9472572160359 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:28:28.469009 17095 consensus_queue.cc:260] T 0f822150741f4ec1a3a9472572160359 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:28:28.469913 17095 ts_tablet_manager.cc:1398] T 0f822150741f4ec1a3a9472572160359 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:28:28.470516 4727 tablet_copy_service.cc:342] P 398c3c8f3195492c900439ad5e1931bb: Request end of tablet copy session 39981a8dfe94462b9a73ba489af4c714-0f822150741f4ec1a3a9472572160359 received from {username='slave'} at 127.29.82.69:46901
I20240502 21:28:28.470628 4727 tablet_copy_service.cc:433] P 398c3c8f3195492c900439ad5e1931bb: ending tablet copy session 39981a8dfe94462b9a73ba489af4c714-0f822150741f4ec1a3a9472572160359 on tablet 0f822150741f4ec1a3a9472572160359 with peer 39981a8dfe94462b9a73ba489af4c714
I20240502 21:28:28.512569 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:28.513499 7617 raft_consensus.cc:1203] T 20b714b400824ecf86890f92aeff19be P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:28.523032 17102 tablet_bootstrap.cc:492] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 31a72286397c4edc9a63eca3412be168: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:28.523466 17102 tablet_bootstrap.cc:492] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 31a72286397c4edc9a63eca3412be168: Bootstrap complete.
I20240502 21:28:28.523773 17102 ts_tablet_manager.cc:1367] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.172s user 0.051s sys 0.008s
I20240502 21:28:28.524466 17102 raft_consensus.cc:348] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:28:28.524698 17102 raft_consensus.cc:729] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:28:28.531809 17102 consensus_queue.cc:260] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:28:28.534791 17102 ts_tablet_manager.cc:1398] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 21:28:28.542140 17097 tablet_bootstrap.cc:492] T 94e66998ede944a6b00478270029ebcf P 398c3c8f3195492c900439ad5e1931bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:28.542498 17097 tablet_bootstrap.cc:492] T 94e66998ede944a6b00478270029ebcf P 398c3c8f3195492c900439ad5e1931bb: Bootstrap complete.
I20240502 21:28:28.542726 17097 ts_tablet_manager.cc:1367] T 94e66998ede944a6b00478270029ebcf P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.188s user 0.009s sys 0.013s
I20240502 21:28:28.543288 17097 raft_consensus.cc:348] T 94e66998ede944a6b00478270029ebcf P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:28.543470 17097 raft_consensus.cc:729] T 94e66998ede944a6b00478270029ebcf P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:28:28.543725 17097 consensus_queue.cc:260] T 94e66998ede944a6b00478270029ebcf P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:28.545815 17097 ts_tablet_manager.cc:1398] T 94e66998ede944a6b00478270029ebcf P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240502 21:28:28.551796 7236 tablet_copy_service.cc:342] P 39981a8dfe94462b9a73ba489af4c714: Request end of tablet copy session 398c3c8f3195492c900439ad5e1931bb-94e66998ede944a6b00478270029ebcf received from {username='slave'} at 127.29.82.67:59759
I20240502 21:28:28.551930 7236 tablet_copy_service.cc:433] P 39981a8dfe94462b9a73ba489af4c714: ending tablet copy session 398c3c8f3195492c900439ad5e1931bb-94e66998ede944a6b00478270029ebcf on tablet 94e66998ede944a6b00478270029ebcf with peer 398c3c8f3195492c900439ad5e1931bb
I20240502 21:28:28.568250 4859 tablet_copy_service.cc:342] P 73a1f2ca1f544de9b0e3b37015f85328: Request end of tablet copy session 31a72286397c4edc9a63eca3412be168-1d0fcf18fd154a72a3bb2f0744febfc7 received from {username='slave'} at 127.29.82.71:36305
I20240502 21:28:28.568398 4859 tablet_copy_service.cc:433] P 73a1f2ca1f544de9b0e3b37015f85328: ending tablet copy session 31a72286397c4edc9a63eca3412be168-1d0fcf18fd154a72a3bb2f0744febfc7 on tablet 1d0fcf18fd154a72a3bb2f0744febfc7 with peer 31a72286397c4edc9a63eca3412be168
I20240502 21:28:28.583274 17124 tablet_copy_client.cc:308] T e95d3785435e450688d87a0ce02d0a34 P 31a72286397c4edc9a63eca3412be168: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.68:42449
I20240502 21:28:28.583993 4859 tablet_copy_service.cc:140] P 73a1f2ca1f544de9b0e3b37015f85328: Received BeginTabletCopySession request for tablet e95d3785435e450688d87a0ce02d0a34 from peer 31a72286397c4edc9a63eca3412be168 ({username='slave'} at 127.29.82.71:36305)
I20240502 21:28:28.584164 4859 tablet_copy_service.cc:161] P 73a1f2ca1f544de9b0e3b37015f85328: Beginning new tablet copy session on tablet e95d3785435e450688d87a0ce02d0a34 from peer 31a72286397c4edc9a63eca3412be168 at {username='slave'} at 127.29.82.71:36305: session id = 31a72286397c4edc9a63eca3412be168-e95d3785435e450688d87a0ce02d0a34
I20240502 21:28:28.587008 4859 tablet_copy_source_session.cc:205] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:28.590416 17124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e95d3785435e450688d87a0ce02d0a34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:28.601877 17124 tablet_copy_client.cc:783] T e95d3785435e450688d87a0ce02d0a34 P 31a72286397c4edc9a63eca3412be168: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:28.602123 17124 tablet_copy_client.cc:647] T e95d3785435e450688d87a0ce02d0a34 P 31a72286397c4edc9a63eca3412be168: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:28.605765 17124 tablet_copy_client.cc:523] T e95d3785435e450688d87a0ce02d0a34 P 31a72286397c4edc9a63eca3412be168: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:28.611474 17124 tablet_bootstrap.cc:492] T e95d3785435e450688d87a0ce02d0a34 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:28:28.626668 4705 raft_consensus.cc:1203] T 94e66998ede944a6b00478270029ebcf P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:28.671975 17124 tablet_bootstrap.cc:492] T e95d3785435e450688d87a0ce02d0a34 P 31a72286397c4edc9a63eca3412be168: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:28.672478 17124 tablet_bootstrap.cc:492] T e95d3785435e450688d87a0ce02d0a34 P 31a72286397c4edc9a63eca3412be168: Bootstrap complete.
I20240502 21:28:28.672770 17124 ts_tablet_manager.cc:1367] T e95d3785435e450688d87a0ce02d0a34 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.061s user 0.052s sys 0.000s
I20240502 21:28:28.673491 17124 raft_consensus.cc:348] T e95d3785435e450688d87a0ce02d0a34 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:28:28.673743 17124 raft_consensus.cc:729] T e95d3785435e450688d87a0ce02d0a34 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:28:28.674019 17124 consensus_queue.cc:260] T e95d3785435e450688d87a0ce02d0a34 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:28:28.674670 17124 ts_tablet_manager.cc:1398] T e95d3785435e450688d87a0ce02d0a34 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:28:28.678413 4859 tablet_copy_service.cc:342] P 73a1f2ca1f544de9b0e3b37015f85328: Request end of tablet copy session 31a72286397c4edc9a63eca3412be168-e95d3785435e450688d87a0ce02d0a34 received from {username='slave'} at 127.29.82.71:36305
I20240502 21:28:28.678547 4859 tablet_copy_service.cc:433] P 73a1f2ca1f544de9b0e3b37015f85328: ending tablet copy session 31a72286397c4edc9a63eca3412be168-e95d3785435e450688d87a0ce02d0a34 on tablet e95d3785435e450688d87a0ce02d0a34 with peer 31a72286397c4edc9a63eca3412be168
I20240502 21:28:28.737967 7213 raft_consensus.cc:1203] T 0f822150741f4ec1a3a9472572160359 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:28.749387 7484 raft_consensus.cc:1203] T e95d3785435e450688d87a0ce02d0a34 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:28:28.755813 7484 raft_consensus.cc:1203] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:28:29.093829 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:29.130538 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:29.163143 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:29.163697 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:29.239413 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:29.245848 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:29.259733 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:29.271802 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:29.586745 7213 consensus_queue.cc:237] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:29.589203 4705 raft_consensus.cc:1261] T 94e66998ede944a6b00478270029ebcf P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:29.589779 7347 raft_consensus.cc:1261] T 94e66998ede944a6b00478270029ebcf P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:29.589835 17134 consensus_queue.cc:1035] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:29.591027 17134 consensus_queue.cc:1035] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:29.601706 17134 raft_consensus.cc:2932] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:29.605521 7350 raft_consensus.cc:2932] T 94e66998ede944a6b00478270029ebcf P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:29.613247 4297 catalog_manager.cc:5484] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: config changed from index 2 to 3, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) evicted. New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:29.645565 4704 raft_consensus.cc:2932] T 94e66998ede944a6b00478270029ebcf P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:29.679499 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet 94e66998ede944a6b00478270029ebcf with delete_type TABLET_DATA_TOMBSTONED (TS 71e1375b799b439fa3c83ca763c039c1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:35984
I20240502 21:28:29.680219 17240 tablet_replica.cc:330] T 94e66998ede944a6b00478270029ebcf P 71e1375b799b439fa3c83ca763c039c1: stopping tablet replica
I20240502 21:28:29.680651 17240 raft_consensus.cc:2229] T 94e66998ede944a6b00478270029ebcf P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:29.680892 17240 raft_consensus.cc:2258] T 94e66998ede944a6b00478270029ebcf P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:29.681604 17240 ts_tablet_manager.cc:1875] T 94e66998ede944a6b00478270029ebcf P 71e1375b799b439fa3c83ca763c039c1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:29.688174 17240 ts_tablet_manager.cc:1888] T 94e66998ede944a6b00478270029ebcf P 71e1375b799b439fa3c83ca763c039c1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:29.688290 17240 log.cc:1196] T 94e66998ede944a6b00478270029ebcf P 71e1375b799b439fa3c83ca763c039c1: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/wals/94e66998ede944a6b00478270029ebcf
I20240502 21:28:29.689147 4284 catalog_manager.cc:4830] TS 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): tablet 94e66998ede944a6b00478270029ebcf (table rebalancer_extra_table_50 [id=f33bc94e9d8b4f11aa285b5624424d4f]) successfully deleted
I20240502 21:28:30.182495 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:30.189821 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:30.206976 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:30.213096 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:30.245441 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:30.250334 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:30.259346 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:30.266161 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:30.442415 7483 consensus_queue.cc:237] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:28:30.458854 7617 raft_consensus.cc:1261] T 20b714b400824ecf86890f92aeff19be P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:30.459117 4439 raft_consensus.cc:1261] T 20b714b400824ecf86890f92aeff19be P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:30.459416 16959 consensus_queue.cc:1035] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:30.459793 17283 consensus_queue.cc:1035] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:30.466576 17283 raft_consensus.cc:2932] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:30.467926 4439 raft_consensus.cc:2932] T 20b714b400824ecf86890f92aeff19be P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:30.476389 7617 raft_consensus.cc:2932] T 20b714b400824ecf86890f92aeff19be P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:30.483963 4294 catalog_manager.cc:5484] T 20b714b400824ecf86890f92aeff19be P 0ddd486109f5434cace13d60d633ac04 reported cstate change: config changed from index 2 to 3, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) evicted. New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:30.517604 4817 tablet_service.cc:1514] Processing DeleteTablet for tablet 20b714b400824ecf86890f92aeff19be with delete_type TABLET_DATA_TOMBSTONED (TS 73a1f2ca1f544de9b0e3b37015f85328 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:56326
I20240502 21:28:30.518458 17351 tablet_replica.cc:330] T 20b714b400824ecf86890f92aeff19be P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:28:30.518769 17351 raft_consensus.cc:2229] T 20b714b400824ecf86890f92aeff19be P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:30.519045 17351 raft_consensus.cc:2258] T 20b714b400824ecf86890f92aeff19be P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:30.519807 17351 ts_tablet_manager.cc:1875] T 20b714b400824ecf86890f92aeff19be P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:30.524569 17351 ts_tablet_manager.cc:1888] T 20b714b400824ecf86890f92aeff19be P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:30.524803 17351 log.cc:1196] T 20b714b400824ecf86890f92aeff19be P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/20b714b400824ecf86890f92aeff19be
I20240502 21:28:30.525624 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet 20b714b400824ecf86890f92aeff19be (table rebalancer_extra_table_50 [id=f33bc94e9d8b4f11aa285b5624424d4f]) successfully deleted
I20240502 21:28:30.847275 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:30.864795 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:30.934443 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling FlushMRSOp(86fa4255c6554ba28e141de3cb328976): perf score=0.033358
I20240502 21:28:30.950423 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling FlushMRSOp(eb81ce7407ab4c9e919a8a5f1b98d545): perf score=0.033365
I20240502 21:28:30.952147 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:30.958673 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:30.971406 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling FlushMRSOp(86fa4255c6554ba28e141de3cb328976): perf score=0.033375
I20240502 21:28:31.008221 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:31.007511 4490 maintenance_manager.cc:419] P 71e1375b799b439fa3c83ca763c039c1: Scheduling FlushMRSOp(e528100a026a4b429ee146912d08a20e): perf score=0.033382
I20240502 21:28:31.045075 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240502 21:28:31.078958 4773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.67:34537, user_credentials={real_user=slave}} blocked reactor thread for 197460us
I20240502 21:28:31.086745 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:31.123471 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:31.280573 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: FlushMRSOp(eb81ce7407ab4c9e919a8a5f1b98d545) complete. Timing: real 0.329s user 0.076s sys 0.031s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":205421,"dirs.queue_time_us":6308,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":5025,"drs_written":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1100,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":218525,"threads_started":3,"wal-append.queue_time_us":6363}
I20240502 21:28:31.285202 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling UndoDeltaBlockGCOp(eb81ce7407ab4c9e919a8a5f1b98d545): 4925 bytes on disk
I20240502 21:28:31.286273 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: UndoDeltaBlockGCOp(eb81ce7407ab4c9e919a8a5f1b98d545) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4}
I20240502 21:28:31.289227 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling FlushMRSOp(e528100a026a4b429ee146912d08a20e): perf score=0.033454
I20240502 21:28:31.382035 4379 maintenance_manager.cc:643] P 71e1375b799b439fa3c83ca763c039c1: FlushMRSOp(e528100a026a4b429ee146912d08a20e) complete. Timing: real 0.371s user 0.068s sys 0.028s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":83251,"dirs.queue_time_us":13790,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":963,"drs_written":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2237,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":106919,"threads_started":3,"wal-append.queue_time_us":10801}
I20240502 21:28:31.390280 4490 maintenance_manager.cc:419] P 71e1375b799b439fa3c83ca763c039c1: Scheduling UndoDeltaBlockGCOp(e528100a026a4b429ee146912d08a20e): 4841 bytes on disk
I20240502 21:28:31.393245 4379 maintenance_manager.cc:643] P 71e1375b799b439fa3c83ca763c039c1: UndoDeltaBlockGCOp(e528100a026a4b429ee146912d08a20e) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4}
I20240502 21:28:31.419778 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: FlushMRSOp(e528100a026a4b429ee146912d08a20e) complete. Timing: real 0.130s user 0.049s sys 0.005s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":11849,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":1842,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1022,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":16256,"threads_started":1,"wal-append.queue_time_us":10317}
I20240502 21:28:31.431033 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling UndoDeltaBlockGCOp(e528100a026a4b429ee146912d08a20e): 4841 bytes on disk
I20240502 21:28:31.432473 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: UndoDeltaBlockGCOp(e528100a026a4b429ee146912d08a20e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4}
I20240502 21:28:31.619555 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: FlushMRSOp(86fa4255c6554ba28e141de3cb328976) complete. Timing: real 0.631s user 0.089s sys 0.034s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":5919,"dirs.queue_time_us":82398,"dirs.run_cpu_time_us":235,"dirs.run_wall_time_us":913,"drs_written":1,"lbm_read_time_us":151,"lbm_reads_lt_1ms":4,"lbm_write_time_us":981,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":131977,"threads_started":3,"wal-append.queue_time_us":40151}
I20240502 21:28:31.642906 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling UndoDeltaBlockGCOp(86fa4255c6554ba28e141de3cb328976): 4586 bytes on disk
I20240502 21:28:31.651013 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: UndoDeltaBlockGCOp(86fa4255c6554ba28e141de3cb328976) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
I20240502 21:28:31.653928 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling FlushMRSOp(eb81ce7407ab4c9e919a8a5f1b98d545): perf score=0.033555
I20240502 21:28:31.741683 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: FlushMRSOp(86fa4255c6554ba28e141de3cb328976) complete. Timing: real 0.769s user 0.063s sys 0.018s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":61451,"dirs.queue_time_us":52137,"dirs.run_cpu_time_us":223,"dirs.run_wall_time_us":12006,"drs_written":1,"lbm_read_time_us":186,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1091,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":122126,"threads_started":3,"wal-append.queue_time_us":7502}
I20240502 21:28:31.765933 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling UndoDeltaBlockGCOp(86fa4255c6554ba28e141de3cb328976): 4586 bytes on disk
I20240502 21:28:31.774392 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: UndoDeltaBlockGCOp(86fa4255c6554ba28e141de3cb328976) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20240502 21:28:31.793298 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling FlushMRSOp(eb81ce7407ab4c9e919a8a5f1b98d545): perf score=0.033596
I20240502 21:28:31.968050 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: FlushMRSOp(eb81ce7407ab4c9e919a8a5f1b98d545) complete. Timing: real 0.174s user 0.076s sys 0.007s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":1135,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":932,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":8641,"threads_started":1,"wal-append.queue_time_us":8533}
I20240502 21:28:31.971725 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling UndoDeltaBlockGCOp(eb81ce7407ab4c9e919a8a5f1b98d545): 4925 bytes on disk
W20240502 21:28:31.975342 7420 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.66:35781, user_credentials={real_user=slave}} blocked reactor thread for 51431.5us
I20240502 21:28:31.975636 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: UndoDeltaBlockGCOp(eb81ce7407ab4c9e919a8a5f1b98d545) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240502 21:28:31.980835 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling FlushMRSOp(e528100a026a4b429ee146912d08a20e): perf score=0.033647
I20240502 21:28:32.004720 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: FlushMRSOp(eb81ce7407ab4c9e919a8a5f1b98d545) complete. Timing: real 0.345s user 0.051s sys 0.007s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":269,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1147,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":912,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":11281,"threads_started":1,"wal-append.queue_time_us":11431}
I20240502 21:28:32.008111 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling UndoDeltaBlockGCOp(eb81ce7407ab4c9e919a8a5f1b98d545): 4925 bytes on disk
I20240502 21:28:32.009593 4839 consensus_queue.cc:237] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:28:32.011116 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: UndoDeltaBlockGCOp(eb81ce7407ab4c9e919a8a5f1b98d545) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
I20240502 21:28:32.040566 4574 raft_consensus.cc:1261] T 93e0911f163f4629a4df0249c5eb8860 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:32.041759 17417 consensus_queue.cc:1035] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:32.042640 7483 raft_consensus.cc:1261] T 93e0911f163f4629a4df0249c5eb8860 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:32.043155 17417 consensus_queue.cc:1035] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:32.047170 17417 raft_consensus.cc:2932] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:32.053165 7483 raft_consensus.cc:2932] T 93e0911f163f4629a4df0249c5eb8860 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:32.057627 4574 raft_consensus.cc:2932] T 93e0911f163f4629a4df0249c5eb8860 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:32.069808 4300 catalog_manager.cc:5484] T 93e0911f163f4629a4df0249c5eb8860 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:32.123260 7194 tablet_service.cc:1514] Processing DeleteTablet for tablet 93e0911f163f4629a4df0249c5eb8860 with delete_type TABLET_DATA_TOMBSTONED (TS 39981a8dfe94462b9a73ba489af4c714 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44652
I20240502 21:28:32.132661 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: FlushMRSOp(e528100a026a4b429ee146912d08a20e) complete. Timing: real 0.152s user 0.048s sys 0.009s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":356,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1454,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3117,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":11934,"threads_started":1,"wal-append.queue_time_us":11411}
I20240502 21:28:32.154924 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling UndoDeltaBlockGCOp(e528100a026a4b429ee146912d08a20e): 4841 bytes on disk
I20240502 21:28:32.155810 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: UndoDeltaBlockGCOp(e528100a026a4b429ee146912d08a20e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4}
I20240502 21:28:32.158095 17460 tablet_replica.cc:330] T 93e0911f163f4629a4df0249c5eb8860 P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:28:32.158417 17460 raft_consensus.cc:2229] T 93e0911f163f4629a4df0249c5eb8860 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:32.159327 17460 raft_consensus.cc:2258] T 93e0911f163f4629a4df0249c5eb8860 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:32.162045 17460 ts_tablet_manager.cc:1875] T 93e0911f163f4629a4df0249c5eb8860 P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:32.199016 17460 ts_tablet_manager.cc:1888] T 93e0911f163f4629a4df0249c5eb8860 P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:32.199291 17460 log.cc:1196] T 93e0911f163f4629a4df0249c5eb8860 P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/93e0911f163f4629a4df0249c5eb8860
I20240502 21:28:32.200235 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet 93e0911f163f4629a4df0249c5eb8860 (table rebalancer_extra_table_50 [id=f33bc94e9d8b4f11aa285b5624424d4f]) successfully deleted
I20240502 21:28:33.163705 4490 maintenance_manager.cc:419] P 71e1375b799b439fa3c83ca763c039c1: Scheduling FlushMRSOp(b5099940a48f4466aa7c713bb88d108e): perf score=0.033360
W20240502 21:28:33.164060 7420 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.66:35781, user_credentials={real_user=slave}} blocked reactor thread for 116996us
I20240502 21:28:33.173681 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling FlushMRSOp(14bd28b4c6a54d98a18351b60f299a3a): perf score=0.033367
I20240502 21:28:33.222853 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling FlushMRSOp(14bd28b4c6a54d98a18351b60f299a3a): perf score=0.033388
I20240502 21:28:33.288477 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling FlushMRSOp(b5099940a48f4466aa7c713bb88d108e): perf score=0.033403
I20240502 21:28:33.320144 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: FlushMRSOp(14bd28b4c6a54d98a18351b60f299a3a) complete. Timing: real 0.146s user 0.048s sys 0.018s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":14868,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1529,"drs_written":1,"lbm_read_time_us":629,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1022,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":21296,"threads_started":2,"wal-append.queue_time_us":3007}
I20240502 21:28:33.333945 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling UndoDeltaBlockGCOp(14bd28b4c6a54d98a18351b60f299a3a): 4925 bytes on disk
I20240502 21:28:33.334719 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: UndoDeltaBlockGCOp(14bd28b4c6a54d98a18351b60f299a3a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20240502 21:28:33.338474 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling FlushMRSOp(016ad8b6874c4b629882f1243ea1fba2): perf score=0.033408
I20240502 21:28:33.397123 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: FlushMRSOp(14bd28b4c6a54d98a18351b60f299a3a) complete. Timing: real 0.174s user 0.053s sys 0.022s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":37120,"dirs.run_cpu_time_us":237,"dirs.run_wall_time_us":7720,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1001,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":63161,"threads_started":2,"wal-append.queue_time_us":12485}
I20240502 21:28:33.401696 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling UndoDeltaBlockGCOp(14bd28b4c6a54d98a18351b60f299a3a): 4925 bytes on disk
I20240502 21:28:33.410878 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: UndoDeltaBlockGCOp(14bd28b4c6a54d98a18351b60f299a3a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
I20240502 21:28:33.414134 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling FlushMRSOp(b5099940a48f4466aa7c713bb88d108e): perf score=0.033436
I20240502 21:28:33.444090 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: FlushMRSOp(b5099940a48f4466aa7c713bb88d108e) complete. Timing: real 0.155s user 0.029s sys 0.031s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":14012,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":22481,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1022,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":29554,"threads_started":2,"wal-append.queue_time_us":15511}
I20240502 21:28:33.448673 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling UndoDeltaBlockGCOp(b5099940a48f4466aa7c713bb88d108e): 4586 bytes on disk
I20240502 21:28:33.449995 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: UndoDeltaBlockGCOp(b5099940a48f4466aa7c713bb88d108e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":123,"lbm_reads_lt_1ms":4}
I20240502 21:28:33.453016 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling FlushMRSOp(016ad8b6874c4b629882f1243ea1fba2): perf score=0.033441
I20240502 21:28:33.455253 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: FlushMRSOp(016ad8b6874c4b629882f1243ea1fba2) complete. Timing: real 0.116s user 0.043s sys 0.011s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1295,"drs_written":1,"lbm_read_time_us":139,"lbm_reads_lt_1ms":4,"lbm_write_time_us":938,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":6407,"threads_started":1,"wal-append.queue_time_us":6303}
I20240502 21:28:33.459039 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling UndoDeltaBlockGCOp(016ad8b6874c4b629882f1243ea1fba2): 4840 bytes on disk
I20240502 21:28:33.459766 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: UndoDeltaBlockGCOp(016ad8b6874c4b629882f1243ea1fba2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240502 21:28:33.525482 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:33.540150 4379 maintenance_manager.cc:643] P 71e1375b799b439fa3c83ca763c039c1: FlushMRSOp(b5099940a48f4466aa7c713bb88d108e) complete. Timing: real 0.376s user 0.053s sys 0.014s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":31847,"dirs.run_cpu_time_us":279,"dirs.run_wall_time_us":2709,"drs_written":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1534,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":73068,"threads_started":2,"wal-append.queue_time_us":18086}
I20240502 21:28:33.564054 4490 maintenance_manager.cc:419] P 71e1375b799b439fa3c83ca763c039c1: Scheduling FlushMRSOp(14bd28b4c6a54d98a18351b60f299a3a): perf score=0.033468
I20240502 21:28:33.580240 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:33.596019 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: FlushMRSOp(b5099940a48f4466aa7c713bb88d108e) complete. Timing: real 0.182s user 0.073s sys 0.026s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":166,"dirs.run_wall_time_us":1723,"drs_written":1,"lbm_read_time_us":122,"lbm_reads_lt_1ms":4,"lbm_write_time_us":997,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":62476,"threads_started":1,"wal-append.queue_time_us":61102}
I20240502 21:28:33.599749 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling FlushMRSOp(016ad8b6874c4b629882f1243ea1fba2): perf score=0.033481
I20240502 21:28:33.600641 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:33.634660 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: FlushMRSOp(016ad8b6874c4b629882f1243ea1fba2) complete. Timing: real 0.181s user 0.060s sys 0.018s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":1314,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":924,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":50692,"threads_started":1,"wal-append.queue_time_us":20682}
I20240502 21:28:33.637990 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling UndoDeltaBlockGCOp(016ad8b6874c4b629882f1243ea1fba2): 4840 bytes on disk
I20240502 21:28:33.639621 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: UndoDeltaBlockGCOp(016ad8b6874c4b629882f1243ea1fba2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240502 21:28:33.687640 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:33.704370 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:33.728821 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:33.792201 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:33.812824 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:33.814863 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: FlushMRSOp(016ad8b6874c4b629882f1243ea1fba2) complete. Timing: real 0.214s user 0.076s sys 0.067s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":50,"dirs.run_cpu_time_us":160,"dirs.run_wall_time_us":4203,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":850,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":134005,"threads_started":1,"wal-append.queue_time_us":134129}
I20240502 21:28:33.818001 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling UndoDeltaBlockGCOp(016ad8b6874c4b629882f1243ea1fba2): 4840 bytes on disk
I20240502 21:28:33.821177 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: UndoDeltaBlockGCOp(016ad8b6874c4b629882f1243ea1fba2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240502 21:28:33.823738 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling UndoDeltaBlockGCOp(b5099940a48f4466aa7c713bb88d108e): 4586 bytes on disk
I20240502 21:28:33.824324 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: UndoDeltaBlockGCOp(b5099940a48f4466aa7c713bb88d108e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
W20240502 21:28:33.919905 4375 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.71:33913, user_credentials={real_user=slave}} blocked reactor thread for 107419us
I20240502 21:28:33.932981 4379 maintenance_manager.cc:643] P 71e1375b799b439fa3c83ca763c039c1: FlushMRSOp(14bd28b4c6a54d98a18351b60f299a3a) complete. Timing: real 0.369s user 0.074s sys 0.038s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":35694,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":2779,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":923,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":42038,"threads_started":2,"wal-append.queue_time_us":6349}
I20240502 21:28:33.939178 4490 maintenance_manager.cc:419] P 71e1375b799b439fa3c83ca763c039c1: Scheduling UndoDeltaBlockGCOp(14bd28b4c6a54d98a18351b60f299a3a): 4925 bytes on disk
I20240502 21:28:33.939842 4379 maintenance_manager.cc:643] P 71e1375b799b439fa3c83ca763c039c1: UndoDeltaBlockGCOp(14bd28b4c6a54d98a18351b60f299a3a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240502 21:28:33.945974 4490 maintenance_manager.cc:419] P 71e1375b799b439fa3c83ca763c039c1: Scheduling UndoDeltaBlockGCOp(b5099940a48f4466aa7c713bb88d108e): 4586 bytes on disk
I20240502 21:28:33.947656 4379 maintenance_manager.cc:643] P 71e1375b799b439fa3c83ca763c039c1: UndoDeltaBlockGCOp(b5099940a48f4466aa7c713bb88d108e) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":214,"lbm_reads_lt_1ms":4}
I20240502 21:28:34.049942 4706 consensus_queue.cc:237] T 2ae670e3d8ae49f2b8c7df65b45b179e P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:34.052327 7347 raft_consensus.cc:1261] T 2ae670e3d8ae49f2b8c7df65b45b179e P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:28:34.052579 7616 raft_consensus.cc:1261] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:28:34.053277 17462 consensus_queue.cc:1035] T 2ae670e3d8ae49f2b8c7df65b45b179e P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:34.053670 16995 consensus_queue.cc:1035] T 2ae670e3d8ae49f2b8c7df65b45b179e P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:34.061730 17462 raft_consensus.cc:2932] T 2ae670e3d8ae49f2b8c7df65b45b179e P 398c3c8f3195492c900439ad5e1931bb [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:34.064946 7347 raft_consensus.cc:2932] T 2ae670e3d8ae49f2b8c7df65b45b179e P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:34.067631 7616 raft_consensus.cc:2932] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:34.073316 4291 catalog_manager.cc:5484] T 2ae670e3d8ae49f2b8c7df65b45b179e P d0456c5e55a542128688904e003d403c reported cstate change: config changed from index 2 to 4, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New cstate: current_term: 2 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:34.097039 7462 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ae670e3d8ae49f2b8c7df65b45b179e with delete_type TABLET_DATA_TOMBSTONED (TS 31a72286397c4edc9a63eca3412be168 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37182
I20240502 21:28:34.109661 17595 tablet_replica.cc:330] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:28:34.110088 17595 raft_consensus.cc:2229] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:34.110276 17595 raft_consensus.cc:2258] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:34.110991 17595 ts_tablet_manager.cc:1875] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:34.116539 17595 ts_tablet_manager.cc:1888] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240502 21:28:34.116672 17595 log.cc:1196] T 2ae670e3d8ae49f2b8c7df65b45b179e P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/2ae670e3d8ae49f2b8c7df65b45b179e
I20240502 21:28:34.117535 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet 2ae670e3d8ae49f2b8c7df65b45b179e (table rebalancer_extra_table_14 [id=e7b935c5901749939ea064dc2f41c619]) successfully deleted
I20240502 21:28:34.460063 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:34.464579 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:34.517153 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:34.541000 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:34.566181 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240502 21:28:34.572330 4508 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.71:33913, user_credentials={real_user=slave}} blocked reactor thread for 128406us
I20240502 21:28:34.580570 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:34.587061 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:34.610556 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 21:28:34.680763 4508 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.68:42449, user_credentials={real_user=slave}} blocked reactor thread for 75577.5us
I20240502 21:28:34.804430 4839 consensus_queue.cc:237] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:28:34.806439 7483 raft_consensus.cc:1261] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:34.806591 4575 raft_consensus.cc:1261] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:34.806985 16377 consensus_queue.cc:1035] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:34.807742 17489 consensus_queue.cc:1035] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:34.819877 17489 raft_consensus.cc:2932] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:34.823967 4575 raft_consensus.cc:2932] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:34.827189 4294 catalog_manager.cc:5484] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:34.836079 7484 raft_consensus.cc:2932] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:34.859359 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d0fcf18fd154a72a3bb2f0744febfc7 with delete_type TABLET_DATA_TOMBSTONED (TS 398c3c8f3195492c900439ad5e1931bb not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44764
I20240502 21:28:34.871390 17688 tablet_replica.cc:330] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:28:34.871784 17688 raft_consensus.cc:2229] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:34.872294 17688 raft_consensus.cc:2258] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:34.873406 17688 ts_tablet_manager.cc:1875] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:34.884552 17688 ts_tablet_manager.cc:1888] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:28:34.886341 17688 log.cc:1196] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/1d0fcf18fd154a72a3bb2f0744febfc7
I20240502 21:28:34.887238 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet 1d0fcf18fd154a72a3bb2f0744febfc7 (table rebalance_test_table_15 [id=943341d7da6a4071b6595f437f13e626]) successfully deleted
I20240502 21:28:35.282800 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:35.294221 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:35.312369 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:35.315899 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:35.357250 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:35.365545 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:35.375690 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 21:28:35.405753 4375 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.71:33913, user_credentials={real_user=slave}} blocked reactor thread for 112502us
I20240502 21:28:35.423221 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling FlushMRSOp(e95d3785435e450688d87a0ce02d0a34): perf score=0.033339
I20240502 21:28:35.466544 4490 maintenance_manager.cc:419] P 71e1375b799b439fa3c83ca763c039c1: Scheduling FlushMRSOp(91044c2ac27d42f8a804bc5eadb13d8f): perf score=0.033348
I20240502 21:28:35.470757 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:35.519745 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling FlushMRSOp(e95d3785435e450688d87a0ce02d0a34): perf score=0.033364
I20240502 21:28:35.555513 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: FlushMRSOp(e95d3785435e450688d87a0ce02d0a34) complete. Timing: real 0.129s user 0.040s sys 0.010s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":10851,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":16726,"drs_written":1,"lbm_read_time_us":147,"lbm_reads_lt_1ms":4,"lbm_write_time_us":904,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":19067,"threads_started":2,"wal-append.queue_time_us":8166}
I20240502 21:28:35.558907 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling UndoDeltaBlockGCOp(e95d3785435e450688d87a0ce02d0a34): 4925 bytes on disk
I20240502 21:28:35.559624 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: UndoDeltaBlockGCOp(e95d3785435e450688d87a0ce02d0a34) complete. Timing: real 0.000s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4}
I20240502 21:28:35.562422 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling FlushMRSOp(dd1cf8e447e54d6e907ffa0e4681515f): perf score=0.033373
I20240502 21:28:35.600661 4379 maintenance_manager.cc:643] P 71e1375b799b439fa3c83ca763c039c1: FlushMRSOp(91044c2ac27d42f8a804bc5eadb13d8f) complete. Timing: real 0.134s user 0.055s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":14957,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":4187,"drs_written":1,"lbm_read_time_us":137,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1274,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":24665,"threads_started":2,"wal-append.queue_time_us":9708}
I20240502 21:28:35.603371 4490 maintenance_manager.cc:419] P 71e1375b799b439fa3c83ca763c039c1: Scheduling UndoDeltaBlockGCOp(91044c2ac27d42f8a804bc5eadb13d8f): 4840 bytes on disk
I20240502 21:28:35.605947 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling FlushMRSOp(e95d3785435e450688d87a0ce02d0a34): perf score=0.033390
I20240502 21:28:35.617551 4379 maintenance_manager.cc:643] P 71e1375b799b439fa3c83ca763c039c1: UndoDeltaBlockGCOp(91044c2ac27d42f8a804bc5eadb13d8f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":122,"lbm_reads_lt_1ms":4}
I20240502 21:28:35.620267 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: FlushMRSOp(e95d3785435e450688d87a0ce02d0a34) complete. Timing: real 0.100s user 0.038s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":313,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":8443,"drs_written":1,"lbm_read_time_us":1564,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":1127,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":1859,"threads_started":2,"wal-append.queue_time_us":389}
I20240502 21:28:35.622900 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling FlushMRSOp(dd1cf8e447e54d6e907ffa0e4681515f): perf score=0.033388
I20240502 21:28:35.677461 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: FlushMRSOp(dd1cf8e447e54d6e907ffa0e4681515f) complete. Timing: real 0.115s user 0.052s sys 0.027s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":142,"dirs.run_wall_time_us":1308,"drs_written":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4,"lbm_write_time_us":968,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":30440,"threads_started":1,"wal-append.queue_time_us":22220}
I20240502 21:28:35.680814 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling UndoDeltaBlockGCOp(dd1cf8e447e54d6e907ffa0e4681515f): 4586 bytes on disk
I20240502 21:28:35.688043 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: UndoDeltaBlockGCOp(dd1cf8e447e54d6e907ffa0e4681515f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
I20240502 21:28:35.724409 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: FlushMRSOp(dd1cf8e447e54d6e907ffa0e4681515f) complete. Timing: real 0.101s user 0.039s sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1446,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":803,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":5194,"threads_started":1,"wal-append.queue_time_us":4881}
I20240502 21:28:35.727950 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling FlushMRSOp(91044c2ac27d42f8a804bc5eadb13d8f): perf score=0.033416
I20240502 21:28:35.728787 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: FlushMRSOp(e95d3785435e450688d87a0ce02d0a34) complete. Timing: real 0.123s user 0.068s sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":10869,"dirs.run_cpu_time_us":248,"dirs.run_wall_time_us":5738,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":830,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":15574,"threads_started":2,"wal-append.queue_time_us":4643}
I20240502 21:28:35.732651 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling FlushMRSOp(91044c2ac27d42f8a804bc5eadb13d8f): perf score=0.033419
I20240502 21:28:35.806779 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: FlushMRSOp(91044c2ac27d42f8a804bc5eadb13d8f) complete. Timing: real 0.072s user 0.039s sys 0.018s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":166,"dirs.run_wall_time_us":1476,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2443,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":9262,"threads_started":1,"wal-append.queue_time_us":9159}
I20240502 21:28:35.809926 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling UndoDeltaBlockGCOp(e95d3785435e450688d87a0ce02d0a34): 4925 bytes on disk
I20240502 21:28:35.810586 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: UndoDeltaBlockGCOp(e95d3785435e450688d87a0ce02d0a34) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240502 21:28:35.815881 4706 consensus_queue.cc:237] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:28:35.829013 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling UndoDeltaBlockGCOp(91044c2ac27d42f8a804bc5eadb13d8f): 4840 bytes on disk
I20240502 21:28:35.833458 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: UndoDeltaBlockGCOp(91044c2ac27d42f8a804bc5eadb13d8f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":133,"lbm_reads_lt_1ms":4}
I20240502 21:28:35.834966 7213 raft_consensus.cc:1261] T 0f822150741f4ec1a3a9472572160359 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:35.835609 17772 consensus_queue.cc:1035] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:35.843672 7483 raft_consensus.cc:1261] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:35.845664 17462 consensus_queue.cc:1035] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:35.857267 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: FlushMRSOp(91044c2ac27d42f8a804bc5eadb13d8f) complete. Timing: real 0.127s user 0.084s sys 0.015s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":2374,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1195,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":29964,"threads_started":1,"wal-append.queue_time_us":29728}
I20240502 21:28:35.861521 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling UndoDeltaBlockGCOp(e95d3785435e450688d87a0ce02d0a34): 4925 bytes on disk
I20240502 21:28:35.865044 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: UndoDeltaBlockGCOp(e95d3785435e450688d87a0ce02d0a34) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240502 21:28:35.868469 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling UndoDeltaBlockGCOp(91044c2ac27d42f8a804bc5eadb13d8f): 4840 bytes on disk
I20240502 21:28:35.872723 17772 raft_consensus.cc:2932] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:35.873003 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: UndoDeltaBlockGCOp(91044c2ac27d42f8a804bc5eadb13d8f) complete. Timing: real 0.004s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":4019,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3}
I20240502 21:28:35.876060 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling UndoDeltaBlockGCOp(dd1cf8e447e54d6e907ffa0e4681515f): 4586 bytes on disk
I20240502 21:28:35.876634 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: UndoDeltaBlockGCOp(dd1cf8e447e54d6e907ffa0e4681515f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
I20240502 21:28:35.876586 7214 raft_consensus.cc:2932] T 0f822150741f4ec1a3a9472572160359 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:35.898254 4296 catalog_manager.cc:5484] T 0f822150741f4ec1a3a9472572160359 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:35.923943 7483 raft_consensus.cc:2932] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:35.929497 7328 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f822150741f4ec1a3a9472572160359 with delete_type TABLET_DATA_TOMBSTONED (TS d0456c5e55a542128688904e003d403c not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47078
I20240502 21:28:35.943989 17821 tablet_replica.cc:330] T 0f822150741f4ec1a3a9472572160359 P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:28:35.945317 17821 raft_consensus.cc:2229] T 0f822150741f4ec1a3a9472572160359 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:35.945611 17821 raft_consensus.cc:2258] T 0f822150741f4ec1a3a9472572160359 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:35.946394 17821 ts_tablet_manager.cc:1875] T 0f822150741f4ec1a3a9472572160359 P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:35.952899 17821 ts_tablet_manager.cc:1888] T 0f822150741f4ec1a3a9472572160359 P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:35.953219 17821 log.cc:1196] T 0f822150741f4ec1a3a9472572160359 P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/0f822150741f4ec1a3a9472572160359
I20240502 21:28:35.956131 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet 0f822150741f4ec1a3a9472572160359 (table rebalancer_extra_table_8 [id=a99691f2b4cb42afb50b56d30958d085]) successfully deleted
I20240502 21:28:36.232177 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:36.238440 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:36.253538 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:36.253829 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:36.268056 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:36.305429 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:36.310364 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:36.378508 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:36.530086 4839 consensus_queue.cc:237] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:28:36.532774 4705 raft_consensus.cc:1261] T 831d24800da54da9b22f9e1bca474755 P 398c3c8f3195492c900439ad5e1931bb [term 2 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:28:36.532960 7347 raft_consensus.cc:1261] T 831d24800da54da9b22f9e1bca474755 P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:28:36.533365 17489 consensus_queue.cc:1035] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:36.533914 17489 consensus_queue.cc:1035] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:36.544368 17907 raft_consensus.cc:2932] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:36.545480 7347 raft_consensus.cc:2932] T 831d24800da54da9b22f9e1bca474755 P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:36.558823 4299 catalog_manager.cc:5484] T 831d24800da54da9b22f9e1bca474755 P d0456c5e55a542128688904e003d403c reported cstate change: config changed from index 2 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New cstate: current_term: 2 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:36.565936 4705 raft_consensus.cc:2932] T 831d24800da54da9b22f9e1bca474755 P 398c3c8f3195492c900439ad5e1931bb [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:36.586655 7194 tablet_service.cc:1514] Processing DeleteTablet for tablet 831d24800da54da9b22f9e1bca474755 with delete_type TABLET_DATA_TOMBSTONED (TS 39981a8dfe94462b9a73ba489af4c714 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44652
I20240502 21:28:36.587303 17922 tablet_replica.cc:330] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:28:36.587647 17922 raft_consensus.cc:2229] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:36.588176 17922 raft_consensus.cc:2258] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:36.588886 17922 ts_tablet_manager.cc:1875] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:36.594904 17922 ts_tablet_manager.cc:1888] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240502 21:28:36.595101 17922 log.cc:1196] T 831d24800da54da9b22f9e1bca474755 P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/831d24800da54da9b22f9e1bca474755
I20240502 21:28:36.597678 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet 831d24800da54da9b22f9e1bca474755 (table rebalancer_extra_table_46 [id=8781896475b940309fa802266d577ff0]) successfully deleted
I20240502 21:28:36.624697 4575 consensus_queue.cc:237] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:28:36.628513 4705 raft_consensus.cc:1261] T 016ad8b6874c4b629882f1243ea1fba2 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:28:36.628808 4509 consensus_peers.cc:485] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8 -> Peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): Couldn't send request to peer 0ddd486109f5434cace13d60d633ac04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 016ad8b6874c4b629882f1243ea1fba2. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:36.630026 4839 raft_consensus.cc:1261] T 016ad8b6874c4b629882f1243ea1fba2 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:28:36.630671 17661 consensus_queue.cc:1035] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:36.632436 17661 consensus_queue.cc:1035] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:36.646230 17760 raft_consensus.cc:2932] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:36.646880 4839 raft_consensus.cc:2932] T 016ad8b6874c4b629882f1243ea1fba2 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:36.650837 4705 raft_consensus.cc:2932] T 016ad8b6874c4b629882f1243ea1fba2 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:36.658435 4291 catalog_manager.cc:5484] T 016ad8b6874c4b629882f1243ea1fba2 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:36.708878 7213 consensus_queue.cc:237] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:28:36.711405 7347 raft_consensus.cc:1261] T 357690713cd44eb3b2a297bfd6bbc17d P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:28:36.711838 7148 consensus_peers.cc:485] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 357690713cd44eb3b2a297bfd6bbc17d. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:36.712527 7618 raft_consensus.cc:1261] T 357690713cd44eb3b2a297bfd6bbc17d P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:36.712483 17954 consensus_queue.cc:1035] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:36.713503 17636 consensus_queue.cc:1035] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:36.717763 17636 raft_consensus.cc:2932] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:36.718343 7347 raft_consensus.cc:2932] T 357690713cd44eb3b2a297bfd6bbc17d P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:36.725780 4291 catalog_manager.cc:5484] T 357690713cd44eb3b2a297bfd6bbc17d P d0456c5e55a542128688904e003d403c reported cstate change: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:36.736783 7618 raft_consensus.cc:2932] T 357690713cd44eb3b2a297bfd6bbc17d P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:36.777050 7213 consensus_queue.cc:237] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:28:36.779321 7347 raft_consensus.cc:1261] T 8f579ec914e540b28a1c7c2e42e498e8 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:36.779943 17957 consensus_queue.cc:1035] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:28:36.780252 7151 consensus_peers.cc:485] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f579ec914e540b28a1c7c2e42e498e8. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:36.781457 7618 raft_consensus.cc:1261] T 8f579ec914e540b28a1c7c2e42e498e8 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:36.782131 17957 consensus_queue.cc:1035] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:36.801048 17954 raft_consensus.cc:2932] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:36.803244 7349 raft_consensus.cc:2932] T 8f579ec914e540b28a1c7c2e42e498e8 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:36.810168 7618 raft_consensus.cc:2932] T 8f579ec914e540b28a1c7c2e42e498e8 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:36.813468 4291 catalog_manager.cc:5484] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:36.919128 4705 consensus_queue.cc:237] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
W20240502 21:28:36.921602 4640 consensus_peers.cc:485] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb -> Peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): Couldn't send request to peer 0ddd486109f5434cace13d60d633ac04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 546a832255ad42bebdf8eaa455ba1c3b. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:36.921620 4575 raft_consensus.cc:1261] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:28:36.921820 4838 raft_consensus.cc:1261] T 546a832255ad42bebdf8eaa455ba1c3b P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:28:36.922137 17462 consensus_queue.cc:1035] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:36.923311 17462 consensus_queue.cc:1035] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:36.945353 17462 raft_consensus.cc:2932] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:36.949039 4838 raft_consensus.cc:2932] T 546a832255ad42bebdf8eaa455ba1c3b P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:36.949558 4575 raft_consensus.cc:2932] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:36.951362 4299 catalog_manager.cc:5484] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb reported cstate change: config changed from index -1 to 4, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:36.998500 4838 consensus_queue.cc:237] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:28:37.000931 7483 raft_consensus.cc:1261] T 845233761c2e45b0a23e25107f5461f1 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:28:37.001144 4770 consensus_peers.cc:485] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 845233761c2e45b0a23e25107f5461f1. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:37.001405 17489 consensus_queue.cc:1035] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:37.001703 7347 raft_consensus.cc:1261] T 845233761c2e45b0a23e25107f5461f1 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:37.002184 17489 consensus_queue.cc:1035] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:37.005100 17904 raft_consensus.cc:2932] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:37.005578 7347 raft_consensus.cc:2932] T 845233761c2e45b0a23e25107f5461f1 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:37.008723 7483 raft_consensus.cc:2932] T 845233761c2e45b0a23e25107f5461f1 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:37.011250 4299 catalog_manager.cc:5484] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:37.093940 7347 consensus_queue.cc:237] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
W20240502 21:28:37.097209 7285 consensus_peers.cc:485] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0a5fdbfa16c946a38a90be15079bda0c. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:37.097255 7483 raft_consensus.cc:1261] T 0a5fdbfa16c946a38a90be15079bda0c P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:37.097888 17130 consensus_queue.cc:1035] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:37.100255 7213 raft_consensus.cc:1261] T 0a5fdbfa16c946a38a90be15079bda0c P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:37.100809 18035 consensus_queue.cc:1035] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:37.103452 17130 raft_consensus.cc:2932] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:37.104282 7213 raft_consensus.cc:2932] T 0a5fdbfa16c946a38a90be15079bda0c P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:37.108407 7483 raft_consensus.cc:2932] T 0a5fdbfa16c946a38a90be15079bda0c P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:37.111169 4291 catalog_manager.cc:5484] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c reported cstate change: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:37.146782 7347 consensus_queue.cc:237] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:28:37.152323 7483 raft_consensus.cc:1261] T 448ac663d66e4b008bfd9a6c2213fdc2 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:28:37.152320 7285 consensus_peers.cc:485] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 448ac663d66e4b008bfd9a6c2213fdc2. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:37.152292 4838 raft_consensus.cc:1261] T 448ac663d66e4b008bfd9a6c2213fdc2 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:37.152952 18035 consensus_queue.cc:1035] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:37.153259 18038 consensus_queue.cc:1035] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:37.160589 18035 raft_consensus.cc:2932] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:37.161149 7483 raft_consensus.cc:2932] T 448ac663d66e4b008bfd9a6c2213fdc2 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:37.164530 4838 raft_consensus.cc:2932] T 448ac663d66e4b008bfd9a6c2213fdc2 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:37.167398 4299 catalog_manager.cc:5484] T 448ac663d66e4b008bfd9a6c2213fdc2 P 31a72286397c4edc9a63eca3412be168 reported cstate change: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:37.171725 18057 ts_tablet_manager.cc:902] T 016ad8b6874c4b629882f1243ea1fba2 P 0ddd486109f5434cace13d60d633ac04: Initiating tablet copy from peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:28:37.180979 18057 tablet_copy_client.cc:308] T 016ad8b6874c4b629882f1243ea1fba2 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.66:35781
I20240502 21:28:37.182117 4595 tablet_copy_service.cc:140] P 7a736a8ebb164ce0a051c0fccd76c3b8: Received BeginTabletCopySession request for tablet 016ad8b6874c4b629882f1243ea1fba2 from peer 0ddd486109f5434cace13d60d633ac04 ({username='slave'} at 127.29.82.72:40279)
I20240502 21:28:37.182284 4595 tablet_copy_service.cc:161] P 7a736a8ebb164ce0a051c0fccd76c3b8: Beginning new tablet copy session on tablet 016ad8b6874c4b629882f1243ea1fba2 from peer 0ddd486109f5434cace13d60d633ac04 at {username='slave'} at 127.29.82.72:40279: session id = 0ddd486109f5434cace13d60d633ac04-016ad8b6874c4b629882f1243ea1fba2
I20240502 21:28:37.185600 4595 tablet_copy_source_session.cc:205] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:28:37.187885 18057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 016ad8b6874c4b629882f1243ea1fba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:37.196398 18057 tablet_copy_client.cc:783] T 016ad8b6874c4b629882f1243ea1fba2 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Starting download of 5 data blocks...
I20240502 21:28:37.208093 18057 tablet_copy_client.cc:647] T 016ad8b6874c4b629882f1243ea1fba2 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:37.214887 18057 tablet_copy_client.cc:523] T 016ad8b6874c4b629882f1243ea1fba2 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:37.218842 18057 tablet_bootstrap.cc:492] T 016ad8b6874c4b629882f1243ea1fba2 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:28:37.231935 18074 ts_tablet_manager.cc:902] T 357690713cd44eb3b2a297bfd6bbc17d P 71e1375b799b439fa3c83ca763c039c1: Initiating tablet copy from peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:28:37.233292 18074 tablet_copy_client.cc:308] T 357690713cd44eb3b2a297bfd6bbc17d P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.69:33193
I20240502 21:28:37.234140 7236 tablet_copy_service.cc:140] P 39981a8dfe94462b9a73ba489af4c714: Received BeginTabletCopySession request for tablet 357690713cd44eb3b2a297bfd6bbc17d from peer 71e1375b799b439fa3c83ca763c039c1 ({username='slave'} at 127.29.82.65:42457)
I20240502 21:28:37.234288 7236 tablet_copy_service.cc:161] P 39981a8dfe94462b9a73ba489af4c714: Beginning new tablet copy session on tablet 357690713cd44eb3b2a297bfd6bbc17d from peer 71e1375b799b439fa3c83ca763c039c1 at {username='slave'} at 127.29.82.65:42457: session id = 71e1375b799b439fa3c83ca763c039c1-357690713cd44eb3b2a297bfd6bbc17d
I20240502 21:28:37.235174 18084 ts_tablet_manager.cc:902] T 8f579ec914e540b28a1c7c2e42e498e8 P 7a736a8ebb164ce0a051c0fccd76c3b8: Initiating tablet copy from peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:28:37.237260 7236 tablet_copy_source_session.cc:205] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:37.240183 18074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 357690713cd44eb3b2a297bfd6bbc17d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:37.250531 18084 tablet_copy_client.cc:308] T 8f579ec914e540b28a1c7c2e42e498e8 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.69:33193
I20240502 21:28:37.251250 7236 tablet_copy_service.cc:140] P 39981a8dfe94462b9a73ba489af4c714: Received BeginTabletCopySession request for tablet 8f579ec914e540b28a1c7c2e42e498e8 from peer 7a736a8ebb164ce0a051c0fccd76c3b8 ({username='slave'} at 127.29.82.66:45181)
I20240502 21:28:37.251379 7236 tablet_copy_service.cc:161] P 39981a8dfe94462b9a73ba489af4c714: Beginning new tablet copy session on tablet 8f579ec914e540b28a1c7c2e42e498e8 from peer 7a736a8ebb164ce0a051c0fccd76c3b8 at {username='slave'} at 127.29.82.66:45181: session id = 7a736a8ebb164ce0a051c0fccd76c3b8-8f579ec914e540b28a1c7c2e42e498e8
I20240502 21:28:37.252554 18074 tablet_copy_client.cc:783] T 357690713cd44eb3b2a297bfd6bbc17d P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:37.252835 18074 tablet_copy_client.cc:647] T 357690713cd44eb3b2a297bfd6bbc17d P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:37.253901 7236 tablet_copy_source_session.cc:205] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:37.256899 18084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f579ec914e540b28a1c7c2e42e498e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:37.257699 18074 tablet_copy_client.cc:523] T 357690713cd44eb3b2a297bfd6bbc17d P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:37.257963 18057 tablet_bootstrap.cc:492] T 016ad8b6874c4b629882f1243ea1fba2 P 0ddd486109f5434cace13d60d633ac04: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:37.258340 18057 tablet_bootstrap.cc:492] T 016ad8b6874c4b629882f1243ea1fba2 P 0ddd486109f5434cace13d60d633ac04: Bootstrap complete.
I20240502 21:28:37.258628 18057 ts_tablet_manager.cc:1367] T 016ad8b6874c4b629882f1243ea1fba2 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.040s user 0.022s sys 0.000s
I20240502 21:28:37.259368 18057 raft_consensus.cc:348] T 016ad8b6874c4b629882f1243ea1fba2 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:28:37.259996 18057 raft_consensus.cc:729] T 016ad8b6874c4b629882f1243ea1fba2 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:28:37.260432 18057 consensus_queue.cc:260] T 016ad8b6874c4b629882f1243ea1fba2 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:28:37.262936 18057 ts_tablet_manager.cc:1398] T 016ad8b6874c4b629882f1243ea1fba2 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240502 21:28:37.263617 4595 tablet_copy_service.cc:342] P 7a736a8ebb164ce0a051c0fccd76c3b8: Request end of tablet copy session 0ddd486109f5434cace13d60d633ac04-016ad8b6874c4b629882f1243ea1fba2 received from {username='slave'} at 127.29.82.72:40279
I20240502 21:28:37.263757 4595 tablet_copy_service.cc:433] P 7a736a8ebb164ce0a051c0fccd76c3b8: ending tablet copy session 0ddd486109f5434cace13d60d633ac04-016ad8b6874c4b629882f1243ea1fba2 on tablet 016ad8b6874c4b629882f1243ea1fba2 with peer 0ddd486109f5434cace13d60d633ac04
I20240502 21:28:37.265201 18074 tablet_bootstrap.cc:492] T 357690713cd44eb3b2a297bfd6bbc17d P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:28:37.265501 4573 consensus_queue.cc:237] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:28:37.264650 18084 tablet_copy_client.cc:783] T 8f579ec914e540b28a1c7c2e42e498e8 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:37.273007 18084 tablet_copy_client.cc:647] T 8f579ec914e540b28a1c7c2e42e498e8 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:37.276321 18074 tablet_bootstrap.cc:492] T 357690713cd44eb3b2a297bfd6bbc17d P 71e1375b799b439fa3c83ca763c039c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:37.276671 18074 tablet_bootstrap.cc:492] T 357690713cd44eb3b2a297bfd6bbc17d P 71e1375b799b439fa3c83ca763c039c1: Bootstrap complete.
I20240502 21:28:37.276965 18074 ts_tablet_manager.cc:1367] T 357690713cd44eb3b2a297bfd6bbc17d P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20240502 21:28:37.277637 18074 raft_consensus.cc:348] T 357690713cd44eb3b2a297bfd6bbc17d P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:28:37.277860 18074 raft_consensus.cc:729] T 357690713cd44eb3b2a297bfd6bbc17d P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:28:37.278122 18074 consensus_queue.cc:260] T 357690713cd44eb3b2a297bfd6bbc17d P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
W20240502 21:28:37.278301 4508 consensus_peers.cc:485] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8 -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ffa252784984df1bbb0ce35a26d4865. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:37.278457 7618 raft_consensus.cc:1261] T 8ffa252784984df1bbb0ce35a26d4865 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:37.278822 18074 ts_tablet_manager.cc:1398] T 357690713cd44eb3b2a297bfd6bbc17d P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:28:37.279462 7235 tablet_copy_service.cc:342] P 39981a8dfe94462b9a73ba489af4c714: Request end of tablet copy session 71e1375b799b439fa3c83ca763c039c1-357690713cd44eb3b2a297bfd6bbc17d received from {username='slave'} at 127.29.82.65:42457
I20240502 21:28:37.279618 7235 tablet_copy_service.cc:433] P 39981a8dfe94462b9a73ba489af4c714: ending tablet copy session 71e1375b799b439fa3c83ca763c039c1-357690713cd44eb3b2a297bfd6bbc17d on tablet 357690713cd44eb3b2a297bfd6bbc17d with peer 71e1375b799b439fa3c83ca763c039c1
I20240502 21:28:37.281940 7349 raft_consensus.cc:1261] T 8ffa252784984df1bbb0ce35a26d4865 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:37.282167 17937 consensus_queue.cc:1035] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:37.283633 17937 consensus_queue.cc:1035] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:37.290261 18084 tablet_copy_client.cc:523] T 8f579ec914e540b28a1c7c2e42e498e8 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:37.297291 18084 tablet_bootstrap.cc:492] T 8f579ec914e540b28a1c7c2e42e498e8 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:28:37.304858 17936 raft_consensus.cc:2932] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:37.305509 7618 raft_consensus.cc:2932] T 8ffa252784984df1bbb0ce35a26d4865 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:37.306349 18084 tablet_bootstrap.cc:492] T 8f579ec914e540b28a1c7c2e42e498e8 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:37.306841 18084 tablet_bootstrap.cc:492] T 8f579ec914e540b28a1c7c2e42e498e8 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap complete.
I20240502 21:28:37.307126 18084 ts_tablet_manager.cc:1367] T 8f579ec914e540b28a1c7c2e42e498e8 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240502 21:28:37.307888 18084 raft_consensus.cc:348] T 8f579ec914e540b28a1c7c2e42e498e8 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:28:37.308218 18084 raft_consensus.cc:729] T 8f579ec914e540b28a1c7c2e42e498e8 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:28:37.308558 18084 consensus_queue.cc:260] T 8f579ec914e540b28a1c7c2e42e498e8 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:28:37.309368 18084 ts_tablet_manager.cc:1398] T 8f579ec914e540b28a1c7c2e42e498e8 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:28:37.310072 7236 tablet_copy_service.cc:342] P 39981a8dfe94462b9a73ba489af4c714: Request end of tablet copy session 7a736a8ebb164ce0a051c0fccd76c3b8-8f579ec914e540b28a1c7c2e42e498e8 received from {username='slave'} at 127.29.82.66:45181
I20240502 21:28:37.310187 7236 tablet_copy_service.cc:433] P 39981a8dfe94462b9a73ba489af4c714: ending tablet copy session 7a736a8ebb164ce0a051c0fccd76c3b8-8f579ec914e540b28a1c7c2e42e498e8 on tablet 8f579ec914e540b28a1c7c2e42e498e8 with peer 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:28:37.313467 4299 catalog_manager.cc:5484] T 8ffa252784984df1bbb0ce35a26d4865 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:37.309285 7349 raft_consensus.cc:2932] T 8ffa252784984df1bbb0ce35a26d4865 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:37.378351 18057 ts_tablet_manager.cc:902] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04: Initiating tablet copy from peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:28:37.393738 18057 tablet_copy_client.cc:308] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.67:34537
I20240502 21:28:37.394379 4727 tablet_copy_service.cc:140] P 398c3c8f3195492c900439ad5e1931bb: Received BeginTabletCopySession request for tablet 546a832255ad42bebdf8eaa455ba1c3b from peer 0ddd486109f5434cace13d60d633ac04 ({username='slave'} at 127.29.82.72:33287)
I20240502 21:28:37.394516 4727 tablet_copy_service.cc:161] P 398c3c8f3195492c900439ad5e1931bb: Beginning new tablet copy session on tablet 546a832255ad42bebdf8eaa455ba1c3b from peer 0ddd486109f5434cace13d60d633ac04 at {username='slave'} at 127.29.82.72:33287: session id = 0ddd486109f5434cace13d60d633ac04-546a832255ad42bebdf8eaa455ba1c3b
I20240502 21:28:37.398231 4727 tablet_copy_source_session.cc:205] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:37.399730 18057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 546a832255ad42bebdf8eaa455ba1c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:37.410718 18057 tablet_copy_client.cc:783] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:37.410570 4838 consensus_queue.cc:237] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:37.411123 18057 tablet_copy_client.cc:647] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:37.413422 18057 tablet_copy_client.cc:523] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:37.416690 7213 raft_consensus.cc:1261] T 7b5db855b5b24c8b87d65c7973f90bbb P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:37.417408 18057 tablet_bootstrap.cc:492] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:28:37.417531 16377 consensus_queue.cc:1035] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:28:37.417876 4773 consensus_peers.cc:485] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b5db855b5b24c8b87d65c7973f90bbb. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:37.418949 7347 raft_consensus.cc:1261] T 7b5db855b5b24c8b87d65c7973f90bbb P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:37.419603 16377 consensus_queue.cc:1035] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:37.422724 16377 raft_consensus.cc:2932] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:37.426753 7349 raft_consensus.cc:2932] T 7b5db855b5b24c8b87d65c7973f90bbb P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:37.434356 4297 catalog_manager.cc:5484] T 7b5db855b5b24c8b87d65c7973f90bbb P d0456c5e55a542128688904e003d403c reported cstate change: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:37.434546 7213 raft_consensus.cc:2932] T 7b5db855b5b24c8b87d65c7973f90bbb P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:37.474602 7664 maintenance_manager.cc:419] P 0ddd486109f5434cace13d60d633ac04: Scheduling UndoDeltaBlockGCOp(016ad8b6874c4b629882f1243ea1fba2): 4840 bytes on disk
I20240502 21:28:37.478780 18057 tablet_bootstrap.cc:492] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:37.478821 7553 maintenance_manager.cc:643] P 0ddd486109f5434cace13d60d633ac04: UndoDeltaBlockGCOp(016ad8b6874c4b629882f1243ea1fba2) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":134,"lbm_reads_lt_1ms":4}
I20240502 21:28:37.482460 18057 tablet_bootstrap.cc:492] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04: Bootstrap complete.
I20240502 21:28:37.482755 18057 ts_tablet_manager.cc:1367] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.065s user 0.049s sys 0.000s
I20240502 21:28:37.483584 18057 raft_consensus.cc:348] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:28:37.483840 18057 raft_consensus.cc:729] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:28:37.486042 18057 consensus_queue.cc:260] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:28:37.487179 18057 ts_tablet_manager.cc:1398] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:28:37.491365 4727 tablet_copy_service.cc:342] P 398c3c8f3195492c900439ad5e1931bb: Request end of tablet copy session 0ddd486109f5434cace13d60d633ac04-546a832255ad42bebdf8eaa455ba1c3b received from {username='slave'} at 127.29.82.72:33287
I20240502 21:28:37.491518 4727 tablet_copy_service.cc:433] P 398c3c8f3195492c900439ad5e1931bb: ending tablet copy session 0ddd486109f5434cace13d60d633ac04-546a832255ad42bebdf8eaa455ba1c3b on tablet 546a832255ad42bebdf8eaa455ba1c3b with peer 0ddd486109f5434cace13d60d633ac04
I20240502 21:28:37.525290 4706 consensus_queue.cc:237] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:28:37.527873 4838 raft_consensus.cc:1261] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:28:37.527920 4573 raft_consensus.cc:1261] T 5f614ef09d764740926c10618370084f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:28:37.528182 4641 consensus_peers.cc:485] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb -> Peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Couldn't send request to peer 39981a8dfe94462b9a73ba489af4c714. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f614ef09d764740926c10618370084f. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:37.528574 17995 consensus_queue.cc:1035] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:37.528869 17462 consensus_queue.cc:1035] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:37.532682 17462 raft_consensus.cc:2932] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:37.537544 4839 raft_consensus.cc:2932] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:37.544873 4297 catalog_manager.cc:5484] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index -1 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:37.549000 4573 raft_consensus.cc:2932] T 5f614ef09d764740926c10618370084f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:37.597738 18074 ts_tablet_manager.cc:902] T 845233761c2e45b0a23e25107f5461f1 P 71e1375b799b439fa3c83ca763c039c1: Initiating tablet copy from peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:28:37.627070 4439 raft_consensus.cc:1203] T 357690713cd44eb3b2a297bfd6bbc17d P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:37.631706 18074 tablet_copy_client.cc:308] T 845233761c2e45b0a23e25107f5461f1 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.68:42449
I20240502 21:28:37.632383 4859 tablet_copy_service.cc:140] P 73a1f2ca1f544de9b0e3b37015f85328: Received BeginTabletCopySession request for tablet 845233761c2e45b0a23e25107f5461f1 from peer 71e1375b799b439fa3c83ca763c039c1 ({username='slave'} at 127.29.82.65:44725)
I20240502 21:28:37.632544 4859 tablet_copy_service.cc:161] P 73a1f2ca1f544de9b0e3b37015f85328: Beginning new tablet copy session on tablet 845233761c2e45b0a23e25107f5461f1 from peer 71e1375b799b439fa3c83ca763c039c1 at {username='slave'} at 127.29.82.65:44725: session id = 71e1375b799b439fa3c83ca763c039c1-845233761c2e45b0a23e25107f5461f1
I20240502 21:28:37.634699 4859 tablet_copy_source_session.cc:205] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:37.637001 18074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 845233761c2e45b0a23e25107f5461f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:37.637092 18084 ts_tablet_manager.cc:902] T 448ac663d66e4b008bfd9a6c2213fdc2 P 7a736a8ebb164ce0a051c0fccd76c3b8: Initiating tablet copy from peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:28:37.642892 18074 tablet_copy_client.cc:783] T 845233761c2e45b0a23e25107f5461f1 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:37.643086 18074 tablet_copy_client.cc:647] T 845233761c2e45b0a23e25107f5461f1 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:37.645187 18074 tablet_copy_client.cc:523] T 845233761c2e45b0a23e25107f5461f1 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:37.648640 18074 tablet_bootstrap.cc:492] T 845233761c2e45b0a23e25107f5461f1 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:28:37.661830 18074 tablet_bootstrap.cc:492] T 845233761c2e45b0a23e25107f5461f1 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:37.664170 18074 tablet_bootstrap.cc:492] T 845233761c2e45b0a23e25107f5461f1 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap complete.
I20240502 21:28:37.664476 18074 ts_tablet_manager.cc:1367] T 845233761c2e45b0a23e25107f5461f1 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.006s
I20240502 21:28:37.665612 18074 raft_consensus.cc:348] T 845233761c2e45b0a23e25107f5461f1 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:28:37.667958 18074 raft_consensus.cc:729] T 845233761c2e45b0a23e25107f5461f1 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:28:37.668300 18074 consensus_queue.cc:260] T 845233761c2e45b0a23e25107f5461f1 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:28:37.669039 18074 ts_tablet_manager.cc:1398] T 845233761c2e45b0a23e25107f5461f1 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 21:28:37.670032 4859 tablet_copy_service.cc:342] P 73a1f2ca1f544de9b0e3b37015f85328: Request end of tablet copy session 71e1375b799b439fa3c83ca763c039c1-845233761c2e45b0a23e25107f5461f1 received from {username='slave'} at 127.29.82.65:44725
I20240502 21:28:37.670171 4859 tablet_copy_service.cc:433] P 73a1f2ca1f544de9b0e3b37015f85328: ending tablet copy session 71e1375b799b439fa3c83ca763c039c1-845233761c2e45b0a23e25107f5461f1 on tablet 845233761c2e45b0a23e25107f5461f1 with peer 71e1375b799b439fa3c83ca763c039c1
I20240502 21:28:37.678939 18084 tablet_copy_client.cc:308] T 448ac663d66e4b008bfd9a6c2213fdc2 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.70:40719
I20240502 21:28:37.679874 7370 tablet_copy_service.cc:140] P d0456c5e55a542128688904e003d403c: Received BeginTabletCopySession request for tablet 448ac663d66e4b008bfd9a6c2213fdc2 from peer 7a736a8ebb164ce0a051c0fccd76c3b8 ({username='slave'} at 127.29.82.66:56529)
I20240502 21:28:37.680087 7370 tablet_copy_service.cc:161] P d0456c5e55a542128688904e003d403c: Beginning new tablet copy session on tablet 448ac663d66e4b008bfd9a6c2213fdc2 from peer 7a736a8ebb164ce0a051c0fccd76c3b8 at {username='slave'} at 127.29.82.66:56529: session id = 7a736a8ebb164ce0a051c0fccd76c3b8-448ac663d66e4b008bfd9a6c2213fdc2
I20240502 21:28:37.683864 7370 tablet_copy_source_session.cc:205] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:37.686257 18165 ts_tablet_manager.cc:902] T 8ffa252784984df1bbb0ce35a26d4865 P 73a1f2ca1f544de9b0e3b37015f85328: Initiating tablet copy from peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:28:37.688736 18084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 448ac663d66e4b008bfd9a6c2213fdc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:37.697894 18084 tablet_copy_client.cc:783] T 448ac663d66e4b008bfd9a6c2213fdc2 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:37.699728 18084 tablet_copy_client.cc:647] T 448ac663d66e4b008bfd9a6c2213fdc2 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:37.702697 18084 tablet_copy_client.cc:523] T 448ac663d66e4b008bfd9a6c2213fdc2 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:37.707715 18084 tablet_bootstrap.cc:492] T 448ac663d66e4b008bfd9a6c2213fdc2 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:28:37.720600 18174 ts_tablet_manager.cc:902] T 0a5fdbfa16c946a38a90be15079bda0c P 398c3c8f3195492c900439ad5e1931bb: Initiating tablet copy from peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:28:37.722865 18084 tablet_bootstrap.cc:492] T 448ac663d66e4b008bfd9a6c2213fdc2 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:37.723493 18084 tablet_bootstrap.cc:492] T 448ac663d66e4b008bfd9a6c2213fdc2 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap complete.
I20240502 21:28:37.724203 18084 ts_tablet_manager.cc:1367] T 448ac663d66e4b008bfd9a6c2213fdc2 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.006s
I20240502 21:28:37.724256 18165 tablet_copy_client.cc:308] T 8ffa252784984df1bbb0ce35a26d4865 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.66:35781
I20240502 21:28:37.725114 18084 raft_consensus.cc:348] T 448ac663d66e4b008bfd9a6c2213fdc2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:28:37.725451 4595 tablet_copy_service.cc:140] P 7a736a8ebb164ce0a051c0fccd76c3b8: Received BeginTabletCopySession request for tablet 8ffa252784984df1bbb0ce35a26d4865 from peer 73a1f2ca1f544de9b0e3b37015f85328 ({username='slave'} at 127.29.82.68:46725)
I20240502 21:28:37.725558 18084 raft_consensus.cc:729] T 448ac663d66e4b008bfd9a6c2213fdc2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:28:37.725598 4595 tablet_copy_service.cc:161] P 7a736a8ebb164ce0a051c0fccd76c3b8: Beginning new tablet copy session on tablet 8ffa252784984df1bbb0ce35a26d4865 from peer 73a1f2ca1f544de9b0e3b37015f85328 at {username='slave'} at 127.29.82.68:46725: session id = 73a1f2ca1f544de9b0e3b37015f85328-8ffa252784984df1bbb0ce35a26d4865
I20240502 21:28:37.725833 18084 consensus_queue.cc:260] T 448ac663d66e4b008bfd9a6c2213fdc2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:28:37.727025 18084 ts_tablet_manager.cc:1398] T 448ac663d66e4b008bfd9a6c2213fdc2 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:28:37.728650 7370 tablet_copy_service.cc:342] P d0456c5e55a542128688904e003d403c: Request end of tablet copy session 7a736a8ebb164ce0a051c0fccd76c3b8-448ac663d66e4b008bfd9a6c2213fdc2 received from {username='slave'} at 127.29.82.66:56529
I20240502 21:28:37.728794 7370 tablet_copy_service.cc:433] P d0456c5e55a542128688904e003d403c: ending tablet copy session 7a736a8ebb164ce0a051c0fccd76c3b8-448ac663d66e4b008bfd9a6c2213fdc2 on tablet 448ac663d66e4b008bfd9a6c2213fdc2 with peer 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:28:37.729022 4595 tablet_copy_source_session.cc:205] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:37.793820 7617 raft_consensus.cc:1203] T 016ad8b6874c4b629882f1243ea1fba2 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:28:37.859495 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling FlushMRSOp(2285212d78b0460ca329949eb3140e69): perf score=0.033345
I20240502 21:28:37.868417 18165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ffa252784984df1bbb0ce35a26d4865. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:37.892927 18165 tablet_copy_client.cc:783] T 8ffa252784984df1bbb0ce35a26d4865 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:37.893335 18165 tablet_copy_client.cc:647] T 8ffa252784984df1bbb0ce35a26d4865 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Starting download of 1 WAL segments...
W20240502 21:28:37.898500 7149 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.72:41223, user_credentials={real_user=slave}} blocked reactor thread for 148149us
I20240502 21:28:37.905963 18165 tablet_copy_client.cc:523] T 8ffa252784984df1bbb0ce35a26d4865 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:37.909894 18165 tablet_bootstrap.cc:492] T 8ffa252784984df1bbb0ce35a26d4865 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:28:37.910326 4490 maintenance_manager.cc:419] P 71e1375b799b439fa3c83ca763c039c1: Scheduling FlushMRSOp(9a3c115189e8442ba7c3fa7f65089d11): perf score=0.033350
I20240502 21:28:37.926048 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling FlushMRSOp(2285212d78b0460ca329949eb3140e69): perf score=0.033359
I20240502 21:28:37.947633 18174 tablet_copy_client.cc:308] T 0a5fdbfa16c946a38a90be15079bda0c P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.70:40719
I20240502 21:28:37.948354 7370 tablet_copy_service.cc:140] P d0456c5e55a542128688904e003d403c: Received BeginTabletCopySession request for tablet 0a5fdbfa16c946a38a90be15079bda0c from peer 398c3c8f3195492c900439ad5e1931bb ({username='slave'} at 127.29.82.67:52175)
I20240502 21:28:37.948503 7370 tablet_copy_service.cc:161] P d0456c5e55a542128688904e003d403c: Beginning new tablet copy session on tablet 0a5fdbfa16c946a38a90be15079bda0c from peer 398c3c8f3195492c900439ad5e1931bb at {username='slave'} at 127.29.82.67:52175: session id = 398c3c8f3195492c900439ad5e1931bb-0a5fdbfa16c946a38a90be15079bda0c
I20240502 21:28:37.951427 7370 tablet_copy_source_session.cc:205] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:37.953188 18174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5fdbfa16c946a38a90be15079bda0c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 21:28:37.953387 7151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.67:34537, user_credentials={real_user=slave}} blocked reactor thread for 160097us
W20240502 21:28:37.953510 4377 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.70:40719, user_credentials={real_user=slave}} blocked reactor thread for 98502.6us
I20240502 21:28:37.957916 4575 raft_consensus.cc:1203] T 8f579ec914e540b28a1c7c2e42e498e8 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:37.970052 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling FlushMRSOp(9a3c115189e8442ba7c3fa7f65089d11): perf score=0.033369
I20240502 21:28:37.974032 18174 tablet_copy_client.cc:783] T 0a5fdbfa16c946a38a90be15079bda0c P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:37.974267 18174 tablet_copy_client.cc:647] T 0a5fdbfa16c946a38a90be15079bda0c P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:37.976650 18174 tablet_copy_client.cc:523] T 0a5fdbfa16c946a38a90be15079bda0c P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:37.977613 7617 raft_consensus.cc:1203] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:28:37.993749 18174 tablet_bootstrap.cc:492] T 0a5fdbfa16c946a38a90be15079bda0c P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:28:38.056041 18204 ts_tablet_manager.cc:902] T 7b5db855b5b24c8b87d65c7973f90bbb P 398c3c8f3195492c900439ad5e1931bb: Initiating tablet copy from peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:28:38.057812 18174 tablet_bootstrap.cc:492] T 0a5fdbfa16c946a38a90be15079bda0c P 398c3c8f3195492c900439ad5e1931bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:38.060803 4573 raft_consensus.cc:1203] T 448ac663d66e4b008bfd9a6c2213fdc2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:38.066067 18165 tablet_bootstrap.cc:492] T 8ffa252784984df1bbb0ce35a26d4865 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:38.066560 18165 tablet_bootstrap.cc:492] T 8ffa252784984df1bbb0ce35a26d4865 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap complete.
I20240502 21:28:38.066841 18165 ts_tablet_manager.cc:1367] T 8ffa252784984df1bbb0ce35a26d4865 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.157s user 0.014s sys 0.009s
I20240502 21:28:38.068472 18174 tablet_bootstrap.cc:492] T 0a5fdbfa16c946a38a90be15079bda0c P 398c3c8f3195492c900439ad5e1931bb: Bootstrap complete.
I20240502 21:28:38.068740 18174 ts_tablet_manager.cc:1367] T 0a5fdbfa16c946a38a90be15079bda0c P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.075s user 0.007s sys 0.003s
I20240502 21:28:38.072221 18174 raft_consensus.cc:348] T 0a5fdbfa16c946a38a90be15079bda0c P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:38.072463 18174 raft_consensus.cc:729] T 0a5fdbfa16c946a38a90be15079bda0c P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:28:38.073882 18174 consensus_queue.cc:260] T 0a5fdbfa16c946a38a90be15079bda0c P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:38.081269 18174 ts_tablet_manager.cc:1398] T 0a5fdbfa16c946a38a90be15079bda0c P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 21:28:38.082026 7370 tablet_copy_service.cc:342] P d0456c5e55a542128688904e003d403c: Request end of tablet copy session 398c3c8f3195492c900439ad5e1931bb-0a5fdbfa16c946a38a90be15079bda0c received from {username='slave'} at 127.29.82.67:52175
I20240502 21:28:38.082142 7370 tablet_copy_service.cc:433] P d0456c5e55a542128688904e003d403c: ending tablet copy session 398c3c8f3195492c900439ad5e1931bb-0a5fdbfa16c946a38a90be15079bda0c on tablet 0a5fdbfa16c946a38a90be15079bda0c with peer 398c3c8f3195492c900439ad5e1931bb
I20240502 21:28:38.083832 18165 raft_consensus.cc:348] T 8ffa252784984df1bbb0ce35a26d4865 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:28:38.084122 18165 raft_consensus.cc:729] T 8ffa252784984df1bbb0ce35a26d4865 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:28:38.084395 18165 consensus_queue.cc:260] T 8ffa252784984df1bbb0ce35a26d4865 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:28:38.086673 18165 ts_tablet_manager.cc:1398] T 8ffa252784984df1bbb0ce35a26d4865 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20240502 21:28:38.087920 4595 tablet_copy_service.cc:342] P 7a736a8ebb164ce0a051c0fccd76c3b8: Request end of tablet copy session 73a1f2ca1f544de9b0e3b37015f85328-8ffa252784984df1bbb0ce35a26d4865 received from {username='slave'} at 127.29.82.68:46725
I20240502 21:28:38.088071 4595 tablet_copy_service.cc:433] P 7a736a8ebb164ce0a051c0fccd76c3b8: ending tablet copy session 73a1f2ca1f544de9b0e3b37015f85328-8ffa252784984df1bbb0ce35a26d4865 on tablet 8ffa252784984df1bbb0ce35a26d4865 with peer 73a1f2ca1f544de9b0e3b37015f85328
I20240502 21:28:38.089967 4439 raft_consensus.cc:1203] T 845233761c2e45b0a23e25107f5461f1 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:38.098148 4839 raft_consensus.cc:1203] T 8ffa252784984df1bbb0ce35a26d4865 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:38.153177 18217 ts_tablet_manager.cc:902] T 5f614ef09d764740926c10618370084f P 39981a8dfe94462b9a73ba489af4c714: Initiating tablet copy from peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
W20240502 21:28:38.171653 7281 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.71:33913, user_credentials={real_user=slave}} blocked reactor thread for 53459.5us
I20240502 21:28:38.215435 4379 maintenance_manager.cc:643] P 71e1375b799b439fa3c83ca763c039c1: FlushMRSOp(9a3c115189e8442ba7c3fa7f65089d11) complete. Timing: real 0.305s user 0.043s sys 0.024s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":53944,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":3765,"drs_written":1,"lbm_read_time_us":136,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2640,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":104952,"threads_started":2,"wal-append.queue_time_us":41899}
I20240502 21:28:38.218384 4490 maintenance_manager.cc:419] P 71e1375b799b439fa3c83ca763c039c1: Scheduling UndoDeltaBlockGCOp(9a3c115189e8442ba7c3fa7f65089d11): 4586 bytes on disk
I20240502 21:28:38.219161 4379 maintenance_manager.cc:643] P 71e1375b799b439fa3c83ca763c039c1: UndoDeltaBlockGCOp(9a3c115189e8442ba7c3fa7f65089d11) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240502 21:28:38.222129 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: FlushMRSOp(2285212d78b0460ca329949eb3140e69) complete. Timing: real 0.296s user 0.073s sys 0.028s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":23004,"dirs.run_cpu_time_us":299,"dirs.run_wall_time_us":4920,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":857,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":29865,"threads_started":2,"wal-append.queue_time_us":6590}
I20240502 21:28:38.225991 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling UndoDeltaBlockGCOp(2285212d78b0460ca329949eb3140e69): 4925 bytes on disk
I20240502 21:28:38.226557 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: UndoDeltaBlockGCOp(2285212d78b0460ca329949eb3140e69) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240502 21:28:38.229910 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling FlushMRSOp(a33fd7d5a8db41a39ba780e16a84f9b1): perf score=0.033438
I20240502 21:28:38.256242 18204 tablet_copy_client.cc:308] T 7b5db855b5b24c8b87d65c7973f90bbb P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.68:42449
I20240502 21:28:38.257125 4859 tablet_copy_service.cc:140] P 73a1f2ca1f544de9b0e3b37015f85328: Received BeginTabletCopySession request for tablet 7b5db855b5b24c8b87d65c7973f90bbb from peer 398c3c8f3195492c900439ad5e1931bb ({username='slave'} at 127.29.82.67:42171)
I20240502 21:28:38.257330 4859 tablet_copy_service.cc:161] P 73a1f2ca1f544de9b0e3b37015f85328: Beginning new tablet copy session on tablet 7b5db855b5b24c8b87d65c7973f90bbb from peer 398c3c8f3195492c900439ad5e1931bb at {username='slave'} at 127.29.82.67:42171: session id = 398c3c8f3195492c900439ad5e1931bb-7b5db855b5b24c8b87d65c7973f90bbb
I20240502 21:28:38.261669 4859 tablet_copy_source_session.cc:205] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:38.263161 18204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b5db855b5b24c8b87d65c7973f90bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:38.274155 18204 tablet_copy_client.cc:783] T 7b5db855b5b24c8b87d65c7973f90bbb P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:38.274420 18204 tablet_copy_client.cc:647] T 7b5db855b5b24c8b87d65c7973f90bbb P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:38.276363 18204 tablet_copy_client.cc:523] T 7b5db855b5b24c8b87d65c7973f90bbb P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:38.281004 18204 tablet_bootstrap.cc:492] T 7b5db855b5b24c8b87d65c7973f90bbb P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:28:38.294651 4705 raft_consensus.cc:1203] T 0a5fdbfa16c946a38a90be15079bda0c P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:38.300189 18217 tablet_copy_client.cc:308] T 5f614ef09d764740926c10618370084f P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.67:34537
I20240502 21:28:38.300932 4727 tablet_copy_service.cc:140] P 398c3c8f3195492c900439ad5e1931bb: Received BeginTabletCopySession request for tablet 5f614ef09d764740926c10618370084f from peer 39981a8dfe94462b9a73ba489af4c714 ({username='slave'} at 127.29.82.69:46901)
I20240502 21:28:38.301064 4727 tablet_copy_service.cc:161] P 398c3c8f3195492c900439ad5e1931bb: Beginning new tablet copy session on tablet 5f614ef09d764740926c10618370084f from peer 39981a8dfe94462b9a73ba489af4c714 at {username='slave'} at 127.29.82.69:46901: session id = 39981a8dfe94462b9a73ba489af4c714-5f614ef09d764740926c10618370084f
I20240502 21:28:38.303144 4727 tablet_copy_source_session.cc:205] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:38.309594 18217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f614ef09d764740926c10618370084f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:38.310313 18204 tablet_bootstrap.cc:492] T 7b5db855b5b24c8b87d65c7973f90bbb P 398c3c8f3195492c900439ad5e1931bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:38.310812 18204 tablet_bootstrap.cc:492] T 7b5db855b5b24c8b87d65c7973f90bbb P 398c3c8f3195492c900439ad5e1931bb: Bootstrap complete.
I20240502 21:28:38.311106 18204 ts_tablet_manager.cc:1367] T 7b5db855b5b24c8b87d65c7973f90bbb P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.002s
I20240502 21:28:38.314970 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:38.322373 18217 tablet_copy_client.cc:783] T 5f614ef09d764740926c10618370084f P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:38.322623 18217 tablet_copy_client.cc:647] T 5f614ef09d764740926c10618370084f P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:38.327347 18217 tablet_copy_client.cc:523] T 5f614ef09d764740926c10618370084f P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:38.333001 18217 tablet_bootstrap.cc:492] T 5f614ef09d764740926c10618370084f P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:28:38.356849 18204 raft_consensus.cc:348] T 7b5db855b5b24c8b87d65c7973f90bbb P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:38.357177 18204 raft_consensus.cc:729] T 7b5db855b5b24c8b87d65c7973f90bbb P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:28:38.358481 18204 consensus_queue.cc:260] T 7b5db855b5b24c8b87d65c7973f90bbb P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:38.367621 18204 ts_tablet_manager.cc:1398] T 7b5db855b5b24c8b87d65c7973f90bbb P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.056s user 0.002s sys 0.000s
I20240502 21:28:38.417675 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:38.417886 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:38.418339 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: FlushMRSOp(2285212d78b0460ca329949eb3140e69) complete. Timing: real 0.558s user 0.068s sys 0.015s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":10810,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":1997,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1022,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":27105,"threads_started":2,"wal-append.queue_time_us":446}
I20240502 21:28:38.424796 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling FlushMRSOp(a33fd7d5a8db41a39ba780e16a84f9b1): perf score=0.033497
I20240502 21:28:38.462327 4859 tablet_copy_service.cc:342] P 73a1f2ca1f544de9b0e3b37015f85328: Request end of tablet copy session 398c3c8f3195492c900439ad5e1931bb-7b5db855b5b24c8b87d65c7973f90bbb received from {username='slave'} at 127.29.82.67:42171
I20240502 21:28:38.462483 4859 tablet_copy_service.cc:433] P 73a1f2ca1f544de9b0e3b37015f85328: ending tablet copy session 398c3c8f3195492c900439ad5e1931bb-7b5db855b5b24c8b87d65c7973f90bbb on tablet 7b5db855b5b24c8b87d65c7973f90bbb with peer 398c3c8f3195492c900439ad5e1931bb
I20240502 21:28:38.482903 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:38.483124 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:38.511560 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:38.536347 4705 raft_consensus.cc:1203] T 7b5db855b5b24c8b87d65c7973f90bbb P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:38.548297 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: FlushMRSOp(a33fd7d5a8db41a39ba780e16a84f9b1) complete. Timing: real 0.317s user 0.034s sys 0.015s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":46,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":1510,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":917,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":31896,"threads_started":1,"wal-append.queue_time_us":29161}
I20240502 21:28:38.554080 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling UndoDeltaBlockGCOp(a33fd7d5a8db41a39ba780e16a84f9b1): 4840 bytes on disk
I20240502 21:28:38.555177 18217 tablet_bootstrap.cc:492] T 5f614ef09d764740926c10618370084f P 39981a8dfe94462b9a73ba489af4c714: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:38.555721 18217 tablet_bootstrap.cc:492] T 5f614ef09d764740926c10618370084f P 39981a8dfe94462b9a73ba489af4c714: Bootstrap complete.
I20240502 21:28:38.556030 18217 ts_tablet_manager.cc:1367] T 5f614ef09d764740926c10618370084f P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.223s user 0.047s sys 0.008s
I20240502 21:28:38.556890 18217 raft_consensus.cc:348] T 5f614ef09d764740926c10618370084f P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:28:38.557127 18217 raft_consensus.cc:729] T 5f614ef09d764740926c10618370084f P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:28:38.557359 18217 consensus_queue.cc:260] T 5f614ef09d764740926c10618370084f P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:28:38.565573 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:38.572548 18217 ts_tablet_manager.cc:1398] T 5f614ef09d764740926c10618370084f P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240502 21:28:38.573339 4727 tablet_copy_service.cc:342] P 398c3c8f3195492c900439ad5e1931bb: Request end of tablet copy session 39981a8dfe94462b9a73ba489af4c714-5f614ef09d764740926c10618370084f received from {username='slave'} at 127.29.82.69:46901
I20240502 21:28:38.573455 4727 tablet_copy_service.cc:433] P 398c3c8f3195492c900439ad5e1931bb: ending tablet copy session 39981a8dfe94462b9a73ba489af4c714-5f614ef09d764740926c10618370084f on tablet 5f614ef09d764740926c10618370084f with peer 39981a8dfe94462b9a73ba489af4c714
I20240502 21:28:38.578871 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: UndoDeltaBlockGCOp(a33fd7d5a8db41a39ba780e16a84f9b1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
W20240502 21:28:38.683663 4774 tablet_replica.cc:1402] Time spent flushing the Log queue took a long time: real 0.253s user 0.044s sys 0.033s
I20240502 21:28:38.689726 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:38.763386 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: FlushMRSOp(9a3c115189e8442ba7c3fa7f65089d11) complete. Timing: real 0.793s user 0.111s sys 0.069s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":53365,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":48000,"drs_written":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2161,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":6016,"thread_start_us":306344,"threads_started":2,"wal-append.queue_time_us":246893}
I20240502 21:28:38.766722 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling FlushMRSOp(a33fd7d5a8db41a39ba780e16a84f9b1): perf score=0.033587
I20240502 21:28:38.815640 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: FlushMRSOp(a33fd7d5a8db41a39ba780e16a84f9b1) complete. Timing: real 0.390s user 0.039s sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1420,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":959,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":54512,"threads_started":1,"wal-append.queue_time_us":54384}
I20240502 21:28:38.822605 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling UndoDeltaBlockGCOp(2285212d78b0460ca329949eb3140e69): 4925 bytes on disk
I20240502 21:28:38.823333 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: UndoDeltaBlockGCOp(2285212d78b0460ca329949eb3140e69) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240502 21:28:38.827849 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling FlushMRSOp(9a3c115189e8442ba7c3fa7f65089d11): perf score=0.033603
I20240502 21:28:38.848361 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: FlushMRSOp(a33fd7d5a8db41a39ba780e16a84f9b1) complete. Timing: real 0.081s user 0.043s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":69,"dirs.run_cpu_time_us":280,"dirs.run_wall_time_us":1603,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1168,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":24491,"threads_started":1,"wal-append.queue_time_us":24379}
I20240502 21:28:38.866595 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling UndoDeltaBlockGCOp(a33fd7d5a8db41a39ba780e16a84f9b1): 4840 bytes on disk
I20240502 21:28:38.868810 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: UndoDeltaBlockGCOp(a33fd7d5a8db41a39ba780e16a84f9b1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240502 21:28:38.873307 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling UndoDeltaBlockGCOp(9a3c115189e8442ba7c3fa7f65089d11): 4586 bytes on disk
I20240502 21:28:38.875123 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: UndoDeltaBlockGCOp(9a3c115189e8442ba7c3fa7f65089d11) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20240502 21:28:38.952172 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: FlushMRSOp(9a3c115189e8442ba7c3fa7f65089d11) complete. Timing: real 0.124s user 0.064s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1796,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":935,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":3681,"threads_started":1,"wal-append.queue_time_us":3792}
I20240502 21:28:38.955246 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling UndoDeltaBlockGCOp(a33fd7d5a8db41a39ba780e16a84f9b1): 4840 bytes on disk
I20240502 21:28:38.956372 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: UndoDeltaBlockGCOp(a33fd7d5a8db41a39ba780e16a84f9b1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":137,"lbm_reads_lt_1ms":4}
I20240502 21:28:38.958180 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling UndoDeltaBlockGCOp(9a3c115189e8442ba7c3fa7f65089d11): 4586 bytes on disk
I20240502 21:28:38.958667 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: UndoDeltaBlockGCOp(9a3c115189e8442ba7c3fa7f65089d11) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240502 21:28:39.003166 7213 raft_consensus.cc:1203] T 5f614ef09d764740926c10618370084f P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:28:39.350142 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:39.366216 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:39.390403 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:39.433939 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:39.557870 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:39.561578 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:39.610960 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:39.648003 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:39.780799 4837 consensus_queue.cc:237] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:39.785463 7213 raft_consensus.cc:1261] T 7b5db855b5b24c8b87d65c7973f90bbb P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:39.785462 4705 raft_consensus.cc:1261] T 7b5db855b5b24c8b87d65c7973f90bbb P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:39.786134 18326 consensus_queue.cc:1035] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:39.786486 17904 consensus_queue.cc:1035] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:39.799440 18326 raft_consensus.cc:2932] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:39.803778 4705 raft_consensus.cc:2932] T 7b5db855b5b24c8b87d65c7973f90bbb P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:39.813700 4296 catalog_manager.cc:5484] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:39.824147 7213 raft_consensus.cc:2932] T 7b5db855b5b24c8b87d65c7973f90bbb P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:39.842092 7328 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b5db855b5b24c8b87d65c7973f90bbb with delete_type TABLET_DATA_TOMBSTONED (TS d0456c5e55a542128688904e003d403c not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47078
I20240502 21:28:39.842716 18361 tablet_replica.cc:330] T 7b5db855b5b24c8b87d65c7973f90bbb P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:28:39.842918 18361 raft_consensus.cc:2229] T 7b5db855b5b24c8b87d65c7973f90bbb P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:39.843116 18361 raft_consensus.cc:2258] T 7b5db855b5b24c8b87d65c7973f90bbb P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:39.843695 18361 ts_tablet_manager.cc:1875] T 7b5db855b5b24c8b87d65c7973f90bbb P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:39.848551 18361 ts_tablet_manager.cc:1888] T 7b5db855b5b24c8b87d65c7973f90bbb P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:39.848660 18361 log.cc:1196] T 7b5db855b5b24c8b87d65c7973f90bbb P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/7b5db855b5b24c8b87d65c7973f90bbb
I20240502 21:28:39.849890 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet 7b5db855b5b24c8b87d65c7973f90bbb (table rebalancer_extra_table_40 [id=13f15c7aba6346b19da83861cfdb5f07]) successfully deleted
I20240502 21:28:40.201884 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:40.206125 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:40.221951 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:40.227342 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:40.286542 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:40.312995 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:40.318547 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240502 21:28:40.357380 7282 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.72:41223, user_credentials={real_user=slave}} blocked reactor thread for 55222.5us
I20240502 21:28:40.362145 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:40.548835 4573 consensus_queue.cc:237] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:28:40.551151 4839 raft_consensus.cc:1261] T 8ffa252784984df1bbb0ce35a26d4865 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:40.551213 7349 raft_consensus.cc:1261] T 8ffa252784984df1bbb0ce35a26d4865 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:40.551651 18328 consensus_queue.cc:1035] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:40.551913 18441 consensus_queue.cc:1035] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:40.554414 18328 raft_consensus.cc:2932] T 8ffa252784984df1bbb0ce35a26d4865 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:40.555083 7349 raft_consensus.cc:2932] T 8ffa252784984df1bbb0ce35a26d4865 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:40.555272 4839 raft_consensus.cc:2932] T 8ffa252784984df1bbb0ce35a26d4865 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:40.561960 4292 catalog_manager.cc:5484] T 8ffa252784984df1bbb0ce35a26d4865 P d0456c5e55a542128688904e003d403c reported cstate change: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:40.572827 7597 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ffa252784984df1bbb0ce35a26d4865 with delete_type TABLET_DATA_TOMBSTONED (TS 0ddd486109f5434cace13d60d633ac04 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44444
I20240502 21:28:40.579182 18447 tablet_replica.cc:330] T 8ffa252784984df1bbb0ce35a26d4865 P 0ddd486109f5434cace13d60d633ac04: stopping tablet replica
I20240502 21:28:40.579452 18447 raft_consensus.cc:2229] T 8ffa252784984df1bbb0ce35a26d4865 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:40.579744 18447 raft_consensus.cc:2258] T 8ffa252784984df1bbb0ce35a26d4865 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:40.580317 18447 ts_tablet_manager.cc:1875] T 8ffa252784984df1bbb0ce35a26d4865 P 0ddd486109f5434cace13d60d633ac04: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:40.586129 18447 ts_tablet_manager.cc:1888] T 8ffa252784984df1bbb0ce35a26d4865 P 0ddd486109f5434cace13d60d633ac04: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:40.591706 18447 log.cc:1196] T 8ffa252784984df1bbb0ce35a26d4865 P 0ddd486109f5434cace13d60d633ac04: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/wals/8ffa252784984df1bbb0ce35a26d4865
I20240502 21:28:40.592586 4285 catalog_manager.cc:4830] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): tablet 8ffa252784984df1bbb0ce35a26d4865 (table rebalancer_extra_table_36 [id=57ddb6dad3c444239d35a1ed0119bba9]) successfully deleted
I20240502 21:28:40.932548 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling FlushMRSOp(5f614ef09d764740926c10618370084f): perf score=0.033339
I20240502 21:28:40.943862 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:40.975206 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:40.983474 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling FlushMRSOp(5f614ef09d764740926c10618370084f): perf score=0.033352
I20240502 21:28:41.009799 4490 maintenance_manager.cc:419] P 71e1375b799b439fa3c83ca763c039c1: Scheduling FlushMRSOp(a607fad427764c46b56f8963952d7613): perf score=0.033355
I20240502 21:28:41.017028 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:41.055852 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:41.062892 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:41.065199 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:41.098893 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:41.105255 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:41.131239 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling FlushMRSOp(c9fe2e18aa6c407fbbcbb74bf45bd520): perf score=0.033396
W20240502 21:28:41.158342 7420 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.66:35781, user_credentials={real_user=slave}} blocked reactor thread for 81613.8us
I20240502 21:28:41.246985 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: FlushMRSOp(5f614ef09d764740926c10618370084f) complete. Timing: real 0.256s user 0.047s sys 0.030s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":30008,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":8928,"drs_written":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1008,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":79250,"threads_started":2,"wal-append.queue_time_us":47807}
I20240502 21:28:41.254053 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling FlushMRSOp(a607fad427764c46b56f8963952d7613): perf score=0.033420
I20240502 21:28:41.304766 4379 maintenance_manager.cc:643] P 71e1375b799b439fa3c83ca763c039c1: FlushMRSOp(a607fad427764c46b56f8963952d7613) complete. Timing: real 0.295s user 0.077s sys 0.033s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":132687,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":5728,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":824,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":141968,"threads_started":2,"wal-append.queue_time_us":8799}
I20240502 21:28:41.308477 4490 maintenance_manager.cc:419] P 71e1375b799b439fa3c83ca763c039c1: Scheduling UndoDeltaBlockGCOp(a607fad427764c46b56f8963952d7613): 4839 bytes on disk
I20240502 21:28:41.309253 4379 maintenance_manager.cc:643] P 71e1375b799b439fa3c83ca763c039c1: UndoDeltaBlockGCOp(a607fad427764c46b56f8963952d7613) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240502 21:28:41.317524 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: FlushMRSOp(a607fad427764c46b56f8963952d7613) complete. Timing: real 0.063s user 0.044s sys 0.011s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":146,"dirs.run_wall_time_us":3162,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1034,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":768,"thread_start_us":6529,"threads_started":1,"wal-append.queue_time_us":6604}
I20240502 21:28:41.320634 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling UndoDeltaBlockGCOp(a607fad427764c46b56f8963952d7613): 4839 bytes on disk
I20240502 21:28:41.321363 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: UndoDeltaBlockGCOp(a607fad427764c46b56f8963952d7613) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240502 21:28:41.324151 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling UndoDeltaBlockGCOp(5f614ef09d764740926c10618370084f): 4586 bytes on disk
I20240502 21:28:41.324817 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: UndoDeltaBlockGCOp(5f614ef09d764740926c10618370084f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240502 21:28:41.338016 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: FlushMRSOp(c9fe2e18aa6c407fbbcbb74bf45bd520) complete. Timing: real 0.206s user 0.047s sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":12282,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":8568,"drs_written":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4,"lbm_write_time_us":968,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":64020,"threads_started":2,"wal-append.queue_time_us":51494}
I20240502 21:28:41.346218 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling FlushMRSOp(5f614ef09d764740926c10618370084f): perf score=0.033449
I20240502 21:28:41.391105 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: FlushMRSOp(5f614ef09d764740926c10618370084f) complete. Timing: real 0.458s user 0.040s sys 0.005s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3614,"dirs.run_cpu_time_us":251,"dirs.run_wall_time_us":5108,"drs_written":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4,"lbm_write_time_us":788,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":25844,"threads_started":2,"wal-append.queue_time_us":2381}
I20240502 21:28:41.394551 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling UndoDeltaBlockGCOp(5f614ef09d764740926c10618370084f): 4586 bytes on disk
I20240502 21:28:41.395391 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: UndoDeltaBlockGCOp(5f614ef09d764740926c10618370084f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":123,"lbm_reads_lt_1ms":4}
I20240502 21:28:41.397854 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling FlushMRSOp(c9fe2e18aa6c407fbbcbb74bf45bd520): perf score=0.033465
W20240502 21:28:41.562978 4376 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.72:41223, user_credentials={real_user=slave}} blocked reactor thread for 105794us
I20240502 21:28:41.571462 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: FlushMRSOp(5f614ef09d764740926c10618370084f) complete. Timing: real 0.225s user 0.049s sys 0.020s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":307,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":1471,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":751,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":83079,"threads_started":1,"wal-append.queue_time_us":83167}
I20240502 21:28:41.584192 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: FlushMRSOp(c9fe2e18aa6c407fbbcbb74bf45bd520) complete. Timing: real 0.180s user 0.052s sys 0.057s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":236,"dirs.run_wall_time_us":2957,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":725,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":77269,"threads_started":1,"wal-append.queue_time_us":77368}
I20240502 21:28:41.586817 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling FlushMRSOp(a607fad427764c46b56f8963952d7613): perf score=0.033506
I20240502 21:28:41.596128 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling UndoDeltaBlockGCOp(c9fe2e18aa6c407fbbcbb74bf45bd520): 4925 bytes on disk
I20240502 21:28:41.597767 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: UndoDeltaBlockGCOp(c9fe2e18aa6c407fbbcbb74bf45bd520) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":132,"lbm_reads_lt_1ms":4}
I20240502 21:28:41.626958 7347 consensus_queue.cc:237] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:28:41.629487 4573 raft_consensus.cc:1261] T 448ac663d66e4b008bfd9a6c2213fdc2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:41.629519 4839 raft_consensus.cc:1261] T 448ac663d66e4b008bfd9a6c2213fdc2 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:41.630074 18420 consensus_queue.cc:1035] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:41.630374 18422 consensus_queue.cc:1035] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:41.633539 18420 raft_consensus.cc:2932] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:41.634088 4839 raft_consensus.cc:2932] T 448ac663d66e4b008bfd9a6c2213fdc2 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:41.641000 4297 catalog_manager.cc:5484] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c reported cstate change: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:41.661245 4574 raft_consensus.cc:2932] T 448ac663d66e4b008bfd9a6c2213fdc2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:41.681501 7462 tablet_service.cc:1514] Processing DeleteTablet for tablet 448ac663d66e4b008bfd9a6c2213fdc2 with delete_type TABLET_DATA_TOMBSTONED (TS 31a72286397c4edc9a63eca3412be168 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:37182
I20240502 21:28:41.683596 18565 tablet_replica.cc:330] T 448ac663d66e4b008bfd9a6c2213fdc2 P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:28:41.683813 18565 raft_consensus.cc:2229] T 448ac663d66e4b008bfd9a6c2213fdc2 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:41.684010 18565 raft_consensus.cc:2258] T 448ac663d66e4b008bfd9a6c2213fdc2 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:41.684654 18565 ts_tablet_manager.cc:1875] T 448ac663d66e4b008bfd9a6c2213fdc2 P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:41.695807 18565 ts_tablet_manager.cc:1888] T 448ac663d66e4b008bfd9a6c2213fdc2 P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:41.696027 18565 log.cc:1196] T 448ac663d66e4b008bfd9a6c2213fdc2 P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/448ac663d66e4b008bfd9a6c2213fdc2
I20240502 21:28:41.696018 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: FlushMRSOp(a607fad427764c46b56f8963952d7613) complete. Timing: real 0.108s user 0.058s sys 0.018s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":62,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":1432,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":931,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":250,"threads_started":1,"wal-append.queue_time_us":355}
I20240502 21:28:41.697664 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet 448ac663d66e4b008bfd9a6c2213fdc2 (table rebalancer_extra_table_56 [id=9ce8c844ead645739f9a918d2d899852]) successfully deleted
I20240502 21:28:41.699518 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling UndoDeltaBlockGCOp(c9fe2e18aa6c407fbbcbb74bf45bd520): 4925 bytes on disk
I20240502 21:28:41.700310 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: UndoDeltaBlockGCOp(c9fe2e18aa6c407fbbcbb74bf45bd520) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240502 21:28:41.703485 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling UndoDeltaBlockGCOp(a607fad427764c46b56f8963952d7613): 4839 bytes on disk
I20240502 21:28:41.704530 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: UndoDeltaBlockGCOp(a607fad427764c46b56f8963952d7613) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240502 21:28:41.708233 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling UndoDeltaBlockGCOp(5f614ef09d764740926c10618370084f): 4586 bytes on disk
I20240502 21:28:41.708832 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: UndoDeltaBlockGCOp(5f614ef09d764740926c10618370084f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240502 21:28:42.076167 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:42.077507 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:42.122910 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:42.196645 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:42.197032 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 21:28:42.217861 4378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.66:35781, user_credentials={real_user=slave}} blocked reactor thread for 125460us
I20240502 21:28:42.229799 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:42.250627 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:42.251083 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:42.433619 7347 consensus_queue.cc:237] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:42.435765 7483 raft_consensus.cc:1261] T 0a5fdbfa16c946a38a90be15079bda0c P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:42.435878 4705 raft_consensus.cc:1261] T 0a5fdbfa16c946a38a90be15079bda0c P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:42.436518 18420 consensus_queue.cc:1035] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:42.437069 18420 consensus_queue.cc:1035] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:42.440312 18420 raft_consensus.cc:2932] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:42.441004 7483 raft_consensus.cc:2932] T 0a5fdbfa16c946a38a90be15079bda0c P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:42.442514 4706 raft_consensus.cc:2932] T 0a5fdbfa16c946a38a90be15079bda0c P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:42.449326 4298 catalog_manager.cc:5484] T 0a5fdbfa16c946a38a90be15079bda0c P 31a72286397c4edc9a63eca3412be168 reported cstate change: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:42.475694 7194 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a5fdbfa16c946a38a90be15079bda0c with delete_type TABLET_DATA_TOMBSTONED (TS 39981a8dfe94462b9a73ba489af4c714 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44652
I20240502 21:28:42.477780 18664 tablet_replica.cc:330] T 0a5fdbfa16c946a38a90be15079bda0c P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:28:42.478049 18664 raft_consensus.cc:2229] T 0a5fdbfa16c946a38a90be15079bda0c P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:42.478318 18664 raft_consensus.cc:2258] T 0a5fdbfa16c946a38a90be15079bda0c P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:42.479136 18664 ts_tablet_manager.cc:1875] T 0a5fdbfa16c946a38a90be15079bda0c P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:42.487526 18664 ts_tablet_manager.cc:1888] T 0a5fdbfa16c946a38a90be15079bda0c P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:42.487784 18664 log.cc:1196] T 0a5fdbfa16c946a38a90be15079bda0c P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/0a5fdbfa16c946a38a90be15079bda0c
I20240502 21:28:42.488727 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet 0a5fdbfa16c946a38a90be15079bda0c (table rebalancer_extra_table_12 [id=e508630eb48f45f88993a4b8e7fd2ea6]) successfully deleted
I20240502 21:28:42.879993 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:42.900334 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:42.955829 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:42.979516 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:43.005344 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:43.042562 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:43.042670 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:43.051398 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:43.212301 4705 consensus_queue.cc:237] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:28:43.214650 4573 raft_consensus.cc:1261] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:43.215157 7618 raft_consensus.cc:1261] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:43.215162 18476 consensus_queue.cc:1035] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:43.215749 18476 consensus_queue.cc:1035] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:43.219169 18526 raft_consensus.cc:2932] T 546a832255ad42bebdf8eaa455ba1c3b P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:43.219813 4573 raft_consensus.cc:2932] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:43.220127 7618 raft_consensus.cc:2932] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:43.228068 4292 catalog_manager.cc:5484] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04 reported cstate change: config changed from index 4 to 5, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) evicted. New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:43.240419 4817 tablet_service.cc:1514] Processing DeleteTablet for tablet 546a832255ad42bebdf8eaa455ba1c3b with delete_type TABLET_DATA_TOMBSTONED (TS 73a1f2ca1f544de9b0e3b37015f85328 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56326
I20240502 21:28:43.241000 18755 tablet_replica.cc:330] T 546a832255ad42bebdf8eaa455ba1c3b P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:28:43.241250 18755 raft_consensus.cc:2229] T 546a832255ad42bebdf8eaa455ba1c3b P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:43.241562 18755 raft_consensus.cc:2258] T 546a832255ad42bebdf8eaa455ba1c3b P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:43.242363 18755 ts_tablet_manager.cc:1875] T 546a832255ad42bebdf8eaa455ba1c3b P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:43.249323 18755 ts_tablet_manager.cc:1888] T 546a832255ad42bebdf8eaa455ba1c3b P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:28:43.249449 18755 log.cc:1196] T 546a832255ad42bebdf8eaa455ba1c3b P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/546a832255ad42bebdf8eaa455ba1c3b
I20240502 21:28:43.250138 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet 546a832255ad42bebdf8eaa455ba1c3b (table rebalance_test_table_11 [id=6578c35615d249838e00b267cf4291d8]) successfully deleted
I20240502 21:28:43.690516 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:43.700363 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:43.708549 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:43.796365 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:43.799007 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:43.829248 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:43.848551 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:43.854856 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:44.074755 7214 consensus_queue.cc:237] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:28:44.077124 4575 raft_consensus.cc:1261] T 8f579ec914e540b28a1c7c2e42e498e8 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:44.077252 7347 raft_consensus.cc:1261] T 8f579ec914e540b28a1c7c2e42e498e8 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:44.077821 18194 consensus_queue.cc:1035] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:44.078150 18793 consensus_queue.cc:1035] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:44.097054 18194 raft_consensus.cc:2932] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:44.099472 4490 maintenance_manager.cc:419] P 71e1375b799b439fa3c83ca763c039c1: Scheduling FlushMRSOp(292bfad92b14456da85bdc874fb7e06e): perf score=0.033352
I20240502 21:28:44.115646 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling FlushMRSOp(128dc7838caa403282ed2868444be271): perf score=0.033359
I20240502 21:28:44.127230 4295 catalog_manager.cc:5484] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:44.132301 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling FlushMRSOp(292bfad92b14456da85bdc874fb7e06e): perf score=0.033363
I20240502 21:28:44.139039 4575 raft_consensus.cc:2932] T 8f579ec914e540b28a1c7c2e42e498e8 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:44.150121 7347 raft_consensus.cc:2932] T 8f579ec914e540b28a1c7c2e42e498e8 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:44.200881 7597 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f579ec914e540b28a1c7c2e42e498e8 with delete_type TABLET_DATA_TOMBSTONED (TS 0ddd486109f5434cace13d60d633ac04 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44444
I20240502 21:28:44.201126 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: FlushMRSOp(128dc7838caa403282ed2868444be271) complete. Timing: real 0.083s user 0.049s sys 0.013s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":9379,"dirs.run_cpu_time_us":151,"dirs.run_wall_time_us":1122,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":998,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":18343,"threads_started":2,"wal-append.queue_time_us":7994}
I20240502 21:28:44.205251 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling UndoDeltaBlockGCOp(128dc7838caa403282ed2868444be271): 4925 bytes on disk
I20240502 21:28:44.205767 4379 maintenance_manager.cc:643] P 71e1375b799b439fa3c83ca763c039c1: FlushMRSOp(292bfad92b14456da85bdc874fb7e06e) complete. Timing: real 0.106s user 0.042s sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":11231,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":2767,"drs_written":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1048,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":15934,"threads_started":2,"wal-append.queue_time_us":4416}
I20240502 21:28:44.206210 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: UndoDeltaBlockGCOp(128dc7838caa403282ed2868444be271) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4}
I20240502 21:28:44.208592 4490 maintenance_manager.cc:419] P 71e1375b799b439fa3c83ca763c039c1: Scheduling UndoDeltaBlockGCOp(292bfad92b14456da85bdc874fb7e06e): 4586 bytes on disk
I20240502 21:28:44.209797 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling FlushMRSOp(546a832255ad42bebdf8eaa455ba1c3b): perf score=0.033378
I20240502 21:28:44.211311 4379 maintenance_manager.cc:643] P 71e1375b799b439fa3c83ca763c039c1: UndoDeltaBlockGCOp(292bfad92b14456da85bdc874fb7e06e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240502 21:28:44.214238 4490 maintenance_manager.cc:419] P 71e1375b799b439fa3c83ca763c039c1: Scheduling FlushMRSOp(128dc7838caa403282ed2868444be271): perf score=0.033379
I20240502 21:28:44.223968 18857 tablet_replica.cc:330] T 8f579ec914e540b28a1c7c2e42e498e8 P 0ddd486109f5434cace13d60d633ac04: stopping tablet replica
I20240502 21:28:44.224287 18857 raft_consensus.cc:2229] T 8f579ec914e540b28a1c7c2e42e498e8 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:44.224574 18857 raft_consensus.cc:2258] T 8f579ec914e540b28a1c7c2e42e498e8 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:44.225302 18857 ts_tablet_manager.cc:1875] T 8f579ec914e540b28a1c7c2e42e498e8 P 0ddd486109f5434cace13d60d633ac04: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:44.239040 18857 ts_tablet_manager.cc:1888] T 8f579ec914e540b28a1c7c2e42e498e8 P 0ddd486109f5434cace13d60d633ac04: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:44.242352 18857 log.cc:1196] T 8f579ec914e540b28a1c7c2e42e498e8 P 0ddd486109f5434cace13d60d633ac04: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/wals/8f579ec914e540b28a1c7c2e42e498e8
I20240502 21:28:44.244026 4285 catalog_manager.cc:4830] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): tablet 8f579ec914e540b28a1c7c2e42e498e8 (table rebalancer_extra_table_12 [id=e508630eb48f45f88993a4b8e7fd2ea6]) successfully deleted
I20240502 21:28:44.252403 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling FlushMRSOp(546a832255ad42bebdf8eaa455ba1c3b): perf score=0.033397
I20240502 21:28:44.298939 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: FlushMRSOp(546a832255ad42bebdf8eaa455ba1c3b) complete. Timing: real 0.089s user 0.046s sys 0.010s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1552,"drs_written":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4,"lbm_write_time_us":997,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":6799,"threads_started":1,"wal-append.queue_time_us":6685}
I20240502 21:28:44.302470 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling UndoDeltaBlockGCOp(546a832255ad42bebdf8eaa455ba1c3b): 4840 bytes on disk
I20240502 21:28:44.303251 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: UndoDeltaBlockGCOp(546a832255ad42bebdf8eaa455ba1c3b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":127,"lbm_reads_lt_1ms":4}
I20240502 21:28:44.326529 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: FlushMRSOp(292bfad92b14456da85bdc874fb7e06e) complete. Timing: real 0.194s user 0.049s sys 0.016s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":22133,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":2830,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1016,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":5504,"thread_start_us":46004,"threads_started":2,"wal-append.queue_time_us":7234}
I20240502 21:28:44.329766 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling UndoDeltaBlockGCOp(292bfad92b14456da85bdc874fb7e06e): 4586 bytes on disk
I20240502 21:28:44.332350 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: UndoDeltaBlockGCOp(292bfad92b14456da85bdc874fb7e06e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240502 21:28:44.344035 4379 maintenance_manager.cc:643] P 71e1375b799b439fa3c83ca763c039c1: FlushMRSOp(128dc7838caa403282ed2868444be271) complete. Timing: real 0.128s user 0.046s sys 0.007s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":5329,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1094,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":857,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":14124,"threads_started":1,"wal-append.queue_time_us":13736}
I20240502 21:28:44.346696 4490 maintenance_manager.cc:419] P 71e1375b799b439fa3c83ca763c039c1: Scheduling UndoDeltaBlockGCOp(128dc7838caa403282ed2868444be271): 4925 bytes on disk
I20240502 21:28:44.348821 4379 maintenance_manager.cc:643] P 71e1375b799b439fa3c83ca763c039c1: UndoDeltaBlockGCOp(128dc7838caa403282ed2868444be271) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4}
I20240502 21:28:44.388689 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: FlushMRSOp(546a832255ad42bebdf8eaa455ba1c3b) complete. Timing: real 0.136s user 0.060s sys 0.027s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":45477,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1848,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":888,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":40178,"threads_started":2,"wal-append.queue_time_us":32272}
I20240502 21:28:44.422129 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling UndoDeltaBlockGCOp(546a832255ad42bebdf8eaa455ba1c3b): 4840 bytes on disk
I20240502 21:28:44.423264 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: UndoDeltaBlockGCOp(546a832255ad42bebdf8eaa455ba1c3b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240502 21:28:44.428880 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling FlushMRSOp(128dc7838caa403282ed2868444be271): perf score=0.033438
I20240502 21:28:44.518754 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: FlushMRSOp(128dc7838caa403282ed2868444be271) complete. Timing: real 0.090s user 0.044s sys 0.032s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":157,"dirs.run_cpu_time_us":223,"dirs.run_wall_time_us":1599,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":927,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":33505,"threads_started":1,"wal-append.queue_time_us":33627}
I20240502 21:28:44.521924 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling UndoDeltaBlockGCOp(128dc7838caa403282ed2868444be271): 4925 bytes on disk
I20240502 21:28:44.522475 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: UndoDeltaBlockGCOp(128dc7838caa403282ed2868444be271) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240502 21:28:44.685408 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:44.726073 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:44.744305 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:44.786597 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:44.817294 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:44.843861 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:44.869292 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:44.900722 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:45.058408 4575 consensus_queue.cc:237] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:28:45.060524 7618 raft_consensus.cc:1261] T 016ad8b6874c4b629882f1243ea1fba2 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:45.060791 4705 raft_consensus.cc:1261] T 016ad8b6874c4b629882f1243ea1fba2 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:45.061504 18630 consensus_queue.cc:1035] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:45.062579 18328 consensus_queue.cc:1035] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:45.065685 18630 raft_consensus.cc:2932] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:45.066332 7618 raft_consensus.cc:2932] T 016ad8b6874c4b629882f1243ea1fba2 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:45.075016 4705 raft_consensus.cc:2932] T 016ad8b6874c4b629882f1243ea1fba2 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:45.085839 4294 catalog_manager.cc:5484] T 016ad8b6874c4b629882f1243ea1fba2 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: config changed from index 4 to 5, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) evicted. New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:28:45.103816 4817 tablet_service.cc:1514] Processing DeleteTablet for tablet 016ad8b6874c4b629882f1243ea1fba2 with delete_type TABLET_DATA_TOMBSTONED (TS 73a1f2ca1f544de9b0e3b37015f85328 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56326
I20240502 21:28:45.107939 18955 tablet_replica.cc:330] T 016ad8b6874c4b629882f1243ea1fba2 P 73a1f2ca1f544de9b0e3b37015f85328: stopping tablet replica
I20240502 21:28:45.108163 18955 raft_consensus.cc:2229] T 016ad8b6874c4b629882f1243ea1fba2 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:45.108373 18955 raft_consensus.cc:2258] T 016ad8b6874c4b629882f1243ea1fba2 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:45.109252 18955 ts_tablet_manager.cc:1875] T 016ad8b6874c4b629882f1243ea1fba2 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:45.134737 18955 ts_tablet_manager.cc:1888] T 016ad8b6874c4b629882f1243ea1fba2 P 73a1f2ca1f544de9b0e3b37015f85328: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:28:45.134864 18955 log.cc:1196] T 016ad8b6874c4b629882f1243ea1fba2 P 73a1f2ca1f544de9b0e3b37015f85328: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/wals/016ad8b6874c4b629882f1243ea1fba2
I20240502 21:28:45.135947 4284 catalog_manager.cc:4830] TS 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): tablet 016ad8b6874c4b629882f1243ea1fba2 (table rebalance_test_table_3 [id=3c51dfaf2d824f93813d8d39758c9a8c]) successfully deleted
I20240502 21:28:45.439324 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:45.440790 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:45.456182 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:45.467388 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:45.518472 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:45.552793 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:45.555538 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:45.643854 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:45.744437 4837 consensus_queue.cc:237] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:28:45.746474 7482 raft_consensus.cc:1261] T e95d3785435e450688d87a0ce02d0a34 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:45.746496 4706 raft_consensus.cc:1261] T e95d3785435e450688d87a0ce02d0a34 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:45.747287 17904 consensus_queue.cc:1035] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:45.747617 18727 consensus_queue.cc:1035] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:45.751986 18727 raft_consensus.cc:2932] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:45.754455 4705 raft_consensus.cc:2932] T e95d3785435e450688d87a0ce02d0a34 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:45.761659 4298 catalog_manager.cc:5484] T e95d3785435e450688d87a0ce02d0a34 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:45.784492 7483 raft_consensus.cc:2932] T e95d3785435e450688d87a0ce02d0a34 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:45.788666 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet e95d3785435e450688d87a0ce02d0a34 with delete_type TABLET_DATA_TOMBSTONED (TS 7a736a8ebb164ce0a051c0fccd76c3b8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60018
I20240502 21:28:45.793262 19041 tablet_replica.cc:330] T e95d3785435e450688d87a0ce02d0a34 P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:28:45.794030 19041 raft_consensus.cc:2229] T e95d3785435e450688d87a0ce02d0a34 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:45.794332 19041 raft_consensus.cc:2258] T e95d3785435e450688d87a0ce02d0a34 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:45.796447 19041 ts_tablet_manager.cc:1875] T e95d3785435e450688d87a0ce02d0a34 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:45.809769 19041 ts_tablet_manager.cc:1888] T e95d3785435e450688d87a0ce02d0a34 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:28:45.809995 19041 log.cc:1196] T e95d3785435e450688d87a0ce02d0a34 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/e95d3785435e450688d87a0ce02d0a34
I20240502 21:28:45.811079 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet e95d3785435e450688d87a0ce02d0a34 (table rebalance_test_table_5 [id=0b924d712d1446f6b103821acdc47c7b]) successfully deleted
I20240502 21:28:46.207890 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:46.264722 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:46.266639 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:46.271883 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 21:28:46.292486 7151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.67:34537, user_credentials={real_user=slave}} blocked reactor thread for 70632.6us
I20240502 21:28:46.292668 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:46.301954 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling FlushMRSOp(81c2664e2d45448b9e08a6fac1c90a0d): perf score=0.033360
I20240502 21:28:46.318661 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 21:28:46.329267 7150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.70:40719, user_credentials={real_user=slave}} blocked reactor thread for 107156us
I20240502 21:28:46.337826 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling FlushMRSOp(5c5cd4328db94367b4201a49d521a145): perf score=0.033365
I20240502 21:28:46.346863 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:46.358639 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling FlushMRSOp(5c5cd4328db94367b4201a49d521a145): perf score=0.033370
I20240502 21:28:46.376384 4490 maintenance_manager.cc:419] P 71e1375b799b439fa3c83ca763c039c1: Scheduling FlushMRSOp(6687a533112445dc86951fc778d0725b): perf score=0.033376
I20240502 21:28:46.380018 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240502 21:28:46.474067 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.71:33913, user_credentials={real_user=slave}} blocked reactor thread for 61777.2us
I20240502 21:28:46.545456 4379 maintenance_manager.cc:643] P 71e1375b799b439fa3c83ca763c039c1: FlushMRSOp(6687a533112445dc86951fc778d0725b) complete. Timing: real 0.169s user 0.054s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":14421,"dirs.run_cpu_time_us":255,"dirs.run_wall_time_us":10936,"drs_written":1,"lbm_read_time_us":473,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1552,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":39916,"threads_started":2,"wal-append.queue_time_us":25464}
I20240502 21:28:46.548259 4490 maintenance_manager.cc:419] P 71e1375b799b439fa3c83ca763c039c1: Scheduling UndoDeltaBlockGCOp(6687a533112445dc86951fc778d0725b): 4841 bytes on disk
I20240502 21:28:46.548933 4379 maintenance_manager.cc:643] P 71e1375b799b439fa3c83ca763c039c1: UndoDeltaBlockGCOp(6687a533112445dc86951fc778d0725b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240502 21:28:46.579764 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: FlushMRSOp(5c5cd4328db94367b4201a49d521a145) complete. Timing: real 0.242s user 0.035s sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":8835,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":1453,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":9054,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":22997,"threads_started":2,"wal-append.queue_time_us":7860}
I20240502 21:28:46.583384 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling UndoDeltaBlockGCOp(5c5cd4328db94367b4201a49d521a145): 4925 bytes on disk
I20240502 21:28:46.584182 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: UndoDeltaBlockGCOp(5c5cd4328db94367b4201a49d521a145) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4}
I20240502 21:28:46.586907 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling FlushMRSOp(6687a533112445dc86951fc778d0725b): perf score=0.033428
I20240502 21:28:46.591696 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: FlushMRSOp(5c5cd4328db94367b4201a49d521a145) complete. Timing: real 0.232s user 0.049s sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":7627,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":1233,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":899,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":7963,"threads_started":2,"wal-append.queue_time_us":437}
I20240502 21:28:46.594965 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling UndoDeltaBlockGCOp(5c5cd4328db94367b4201a49d521a145): 4925 bytes on disk
I20240502 21:28:46.595786 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: UndoDeltaBlockGCOp(5c5cd4328db94367b4201a49d521a145) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20240502 21:28:46.595887 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: FlushMRSOp(81c2664e2d45448b9e08a6fac1c90a0d) complete. Timing: real 0.294s user 0.043s sys 0.013s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":416,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":1611,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":937,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":3092,"threads_started":2,"wal-append.queue_time_us":2875}
I20240502 21:28:46.598034 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling FlushMRSOp(81c2664e2d45448b9e08a6fac1c90a0d): perf score=0.033425
I20240502 21:28:46.599428 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling FlushMRSOp(6687a533112445dc86951fc778d0725b): perf score=0.033438
I20240502 21:28:46.655364 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: FlushMRSOp(6687a533112445dc86951fc778d0725b) complete. Timing: real 0.068s user 0.039s sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1944,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1041,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":6440,"threads_started":1,"wal-append.queue_time_us":6338}
I20240502 21:28:46.658746 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling UndoDeltaBlockGCOp(6687a533112445dc86951fc778d0725b): 4841 bytes on disk
I20240502 21:28:46.659487 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: UndoDeltaBlockGCOp(6687a533112445dc86951fc778d0725b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240502 21:28:46.663813 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling FlushMRSOp(81c2664e2d45448b9e08a6fac1c90a0d): perf score=0.033445
I20240502 21:28:46.665239 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: FlushMRSOp(6687a533112445dc86951fc778d0725b) complete. Timing: real 0.064s user 0.040s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":129,"dirs.run_wall_time_us":1792,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":897,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":1820,"threads_started":1,"wal-append.queue_time_us":1932}
I20240502 21:28:46.669114 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling FlushMRSOp(5c5cd4328db94367b4201a49d521a145): perf score=0.033453
I20240502 21:28:46.718834 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: FlushMRSOp(5c5cd4328db94367b4201a49d521a145) complete. Timing: real 0.049s user 0.036s sys 0.007s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":41,"dirs.run_cpu_time_us":141,"dirs.run_wall_time_us":1726,"drs_written":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1013,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":3714,"threads_started":1,"wal-append.queue_time_us":3601}
I20240502 21:28:46.722546 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling UndoDeltaBlockGCOp(5c5cd4328db94367b4201a49d521a145): 4925 bytes on disk
I20240502 21:28:46.724190 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: UndoDeltaBlockGCOp(5c5cd4328db94367b4201a49d521a145) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240502 21:28:46.726938 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling UndoDeltaBlockGCOp(6687a533112445dc86951fc778d0725b): 4841 bytes on disk
I20240502 21:28:46.728653 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: UndoDeltaBlockGCOp(6687a533112445dc86951fc778d0725b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240502 21:28:46.742105 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling UndoDeltaBlockGCOp(81c2664e2d45448b9e08a6fac1c90a0d): 4586 bytes on disk
I20240502 21:28:46.742673 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: UndoDeltaBlockGCOp(81c2664e2d45448b9e08a6fac1c90a0d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240502 21:28:46.751123 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: FlushMRSOp(81c2664e2d45448b9e08a6fac1c90a0d) complete. Timing: real 0.087s user 0.049s sys 0.023s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":127,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1667,"drs_written":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1078,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":19911,"threads_started":1,"wal-append.queue_time_us":19771}
I20240502 21:28:46.754575 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling UndoDeltaBlockGCOp(81c2664e2d45448b9e08a6fac1c90a0d): 4586 bytes on disk
I20240502 21:28:46.755446 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: UndoDeltaBlockGCOp(81c2664e2d45448b9e08a6fac1c90a0d) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20240502 21:28:46.776669 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: FlushMRSOp(81c2664e2d45448b9e08a6fac1c90a0d) complete. Timing: real 0.178s user 0.074s sys 0.027s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":10985,"dirs.run_cpu_time_us":146,"dirs.run_wall_time_us":1550,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1096,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":82346,"threads_started":1,"wal-append.queue_time_us":82460}
I20240502 21:28:46.779538 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling UndoDeltaBlockGCOp(81c2664e2d45448b9e08a6fac1c90a0d): 4586 bytes on disk
I20240502 21:28:46.780395 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: UndoDeltaBlockGCOp(81c2664e2d45448b9e08a6fac1c90a0d) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":129,"lbm_reads_lt_1ms":4}
I20240502 21:28:46.820628 4837 consensus_queue.cc:237] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:28:46.822938 4441 raft_consensus.cc:1261] T 845233761c2e45b0a23e25107f5461f1 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:46.827845 7482 raft_consensus.cc:1261] T 845233761c2e45b0a23e25107f5461f1 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:46.829825 19122 consensus_queue.cc:1035] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:46.830359 19122 consensus_queue.cc:1035] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:46.841073 18727 raft_consensus.cc:2932] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:46.844206 4441 raft_consensus.cc:2932] T 845233761c2e45b0a23e25107f5461f1 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:46.851647 4293 catalog_manager.cc:5484] T 845233761c2e45b0a23e25107f5461f1 P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:46.864390 7482 raft_consensus.cc:2932] T 845233761c2e45b0a23e25107f5461f1 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:46.885490 7328 tablet_service.cc:1514] Processing DeleteTablet for tablet 845233761c2e45b0a23e25107f5461f1 with delete_type TABLET_DATA_TOMBSTONED (TS d0456c5e55a542128688904e003d403c not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47078
I20240502 21:28:46.890120 19167 tablet_replica.cc:330] T 845233761c2e45b0a23e25107f5461f1 P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:28:46.890388 19167 raft_consensus.cc:2229] T 845233761c2e45b0a23e25107f5461f1 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:46.890636 19167 raft_consensus.cc:2258] T 845233761c2e45b0a23e25107f5461f1 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:46.891501 19167 ts_tablet_manager.cc:1875] T 845233761c2e45b0a23e25107f5461f1 P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:46.898686 19167 ts_tablet_manager.cc:1888] T 845233761c2e45b0a23e25107f5461f1 P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:46.898821 19167 log.cc:1196] T 845233761c2e45b0a23e25107f5461f1 P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/845233761c2e45b0a23e25107f5461f1
I20240502 21:28:46.899914 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet 845233761c2e45b0a23e25107f5461f1 (table rebalancer_extra_table_4 [id=2ba724ea31944444b73e3c52e98f06e7]) successfully deleted
I20240502 21:28:47.217953 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:47.232213 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:47.232789 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:47.239117 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:47.252632 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:47.276456 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:47.316812 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:47.319783 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:47.520646 7213 consensus_queue.cc:237] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:28:47.522576 7618 raft_consensus.cc:1261] T 357690713cd44eb3b2a297bfd6bbc17d P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:47.522681 4440 raft_consensus.cc:1261] T 357690713cd44eb3b2a297bfd6bbc17d P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:47.523355 19110 consensus_queue.cc:1035] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:47.523707 18924 consensus_queue.cc:1035] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:47.526908 18924 raft_consensus.cc:2932] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:47.527851 7618 raft_consensus.cc:2932] T 357690713cd44eb3b2a297bfd6bbc17d P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:47.531352 4440 raft_consensus.cc:2932] T 357690713cd44eb3b2a297bfd6bbc17d P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:47.533914 4296 catalog_manager.cc:5484] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:47.561192 7328 tablet_service.cc:1514] Processing DeleteTablet for tablet 357690713cd44eb3b2a297bfd6bbc17d with delete_type TABLET_DATA_TOMBSTONED (TS d0456c5e55a542128688904e003d403c not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47078
I20240502 21:28:47.561841 19260 tablet_replica.cc:330] T 357690713cd44eb3b2a297bfd6bbc17d P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:28:47.562094 19260 raft_consensus.cc:2229] T 357690713cd44eb3b2a297bfd6bbc17d P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:47.562357 19260 raft_consensus.cc:2258] T 357690713cd44eb3b2a297bfd6bbc17d P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:47.563082 19260 ts_tablet_manager.cc:1875] T 357690713cd44eb3b2a297bfd6bbc17d P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:47.569167 19260 ts_tablet_manager.cc:1888] T 357690713cd44eb3b2a297bfd6bbc17d P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:47.569293 19260 log.cc:1196] T 357690713cd44eb3b2a297bfd6bbc17d P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/357690713cd44eb3b2a297bfd6bbc17d
I20240502 21:28:47.570268 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet 357690713cd44eb3b2a297bfd6bbc17d (table rebalancer_extra_table_12 [id=e508630eb48f45f88993a4b8e7fd2ea6]) successfully deleted
I20240502 21:28:47.634533 7483 consensus_queue.cc:237] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:28:47.637109 7349 raft_consensus.cc:1261] T 269eb0e719804925961e4bcd17c18596 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:47.637164 7618 raft_consensus.cc:1261] T 269eb0e719804925961e4bcd17c18596 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:47.637706 18798 consensus_queue.cc:1035] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:28:47.638499 7416 consensus_peers.cc:485] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 269eb0e719804925961e4bcd17c18596. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:47.639204 18798 consensus_queue.cc:1035] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:47.655771 18798 raft_consensus.cc:2932] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:47.656219 7349 raft_consensus.cc:2932] T 269eb0e719804925961e4bcd17c18596 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:47.662302 4297 catalog_manager.cc:5484] T 269eb0e719804925961e4bcd17c18596 P d0456c5e55a542128688904e003d403c reported cstate change: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:47.673386 7616 raft_consensus.cc:2932] T 269eb0e719804925961e4bcd17c18596 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:47.726497 4573 consensus_queue.cc:237] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:28:47.728998 4705 raft_consensus.cc:1261] T 5c5cd4328db94367b4201a49d521a145 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:28:47.729116 4837 raft_consensus.cc:1261] T 5c5cd4328db94367b4201a49d521a145 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:28:47.729794 4508 consensus_peers.cc:485] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8 -> Peer 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): Couldn't send request to peer 31a72286397c4edc9a63eca3412be168. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5c5cd4328db94367b4201a49d521a145. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:47.730135 19118 consensus_queue.cc:1035] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:47.730916 19118 consensus_queue.cc:1035] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:47.734285 19226 raft_consensus.cc:2932] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:47.734889 4705 raft_consensus.cc:2932] T 5c5cd4328db94367b4201a49d521a145 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:47.738205 4839 raft_consensus.cc:2932] T 5c5cd4328db94367b4201a49d521a145 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:47.742231 4291 catalog_manager.cc:5484] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: config changed from index -1 to 4, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:47.807752 7349 consensus_queue.cc:237] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:47.810674 7214 raft_consensus.cc:1261] T 26a00b5700a54672b7389e87c2a4dfbd P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:47.811250 4573 raft_consensus.cc:1261] T 26a00b5700a54672b7389e87c2a4dfbd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:47.811785 18982 consensus_queue.cc:1035] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:28:47.812301 7285 consensus_peers.cc:485] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26a00b5700a54672b7389e87c2a4dfbd. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:47.813961 18982 consensus_queue.cc:1035] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:47.821609 19131 raft_consensus.cc:2932] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:47.826447 7214 raft_consensus.cc:2932] T 26a00b5700a54672b7389e87c2a4dfbd P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:47.828846 4297 catalog_manager.cc:5484] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c reported cstate change: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:47.832849 4573 raft_consensus.cc:2932] T 26a00b5700a54672b7389e87c2a4dfbd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:47.937700 7214 consensus_queue.cc:237] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:28:47.940358 4839 raft_consensus.cc:1261] T e6f70f1bafd44613abb202a37d4c8b64 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:28:47.940560 7150 consensus_peers.cc:485] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 -> Peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Couldn't send request to peer d0456c5e55a542128688904e003d403c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6f70f1bafd44613abb202a37d4c8b64. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:47.941001 19110 consensus_queue.cc:1035] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:47.941182 7483 raft_consensus.cc:1261] T e6f70f1bafd44613abb202a37d4c8b64 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:47.941867 19110 consensus_queue.cc:1035] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:47.953783 18924 raft_consensus.cc:2932] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:47.954454 4839 raft_consensus.cc:2932] T e6f70f1bafd44613abb202a37d4c8b64 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:47.958310 7483 raft_consensus.cc:2932] T e6f70f1bafd44613abb202a37d4c8b64 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:47.960762 4291 catalog_manager.cc:5484] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: config changed from index -1 to 2, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:47.997387 4573 consensus_queue.cc:237] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:28:47.999470 4705 raft_consensus.cc:1261] T a33fd7d5a8db41a39ba780e16a84f9b1 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:28:47.999697 4510 consensus_peers.cc:485] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8 -> Peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Couldn't send request to peer 39981a8dfe94462b9a73ba489af4c714. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a33fd7d5a8db41a39ba780e16a84f9b1. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:47.999754 4839 raft_consensus.cc:1261] T a33fd7d5a8db41a39ba780e16a84f9b1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:28:48.000063 19226 consensus_queue.cc:1035] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:48.000352 19118 consensus_queue.cc:1035] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:48.002885 19226 raft_consensus.cc:2932] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:48.003460 4839 raft_consensus.cc:2932] T a33fd7d5a8db41a39ba780e16a84f9b1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:48.011679 4705 raft_consensus.cc:2932] T a33fd7d5a8db41a39ba780e16a84f9b1 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:48.016542 4297 catalog_manager.cc:5484] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: config changed from index -1 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:48.054636 19384 ts_tablet_manager.cc:902] T 269eb0e719804925961e4bcd17c18596 P 71e1375b799b439fa3c83ca763c039c1: Initiating tablet copy from peer 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:28:48.073083 19384 tablet_copy_client.cc:308] T 269eb0e719804925961e4bcd17c18596 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.71:33913
I20240502 21:28:48.073805 7505 tablet_copy_service.cc:140] P 31a72286397c4edc9a63eca3412be168: Received BeginTabletCopySession request for tablet 269eb0e719804925961e4bcd17c18596 from peer 71e1375b799b439fa3c83ca763c039c1 ({username='slave'} at 127.29.82.65:39791)
I20240502 21:28:48.073982 7505 tablet_copy_service.cc:161] P 31a72286397c4edc9a63eca3412be168: Beginning new tablet copy session on tablet 269eb0e719804925961e4bcd17c18596 from peer 71e1375b799b439fa3c83ca763c039c1 at {username='slave'} at 127.29.82.65:39791: session id = 71e1375b799b439fa3c83ca763c039c1-269eb0e719804925961e4bcd17c18596
I20240502 21:28:48.076845 7505 tablet_copy_source_session.cc:205] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:48.078223 19384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 269eb0e719804925961e4bcd17c18596. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:48.084877 19384 tablet_copy_client.cc:783] T 269eb0e719804925961e4bcd17c18596 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:48.085116 19384 tablet_copy_client.cc:647] T 269eb0e719804925961e4bcd17c18596 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:48.086858 19384 tablet_copy_client.cc:523] T 269eb0e719804925961e4bcd17c18596 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:48.092212 19384 tablet_bootstrap.cc:492] T 269eb0e719804925961e4bcd17c18596 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:28:48.105175 19384 tablet_bootstrap.cc:492] T 269eb0e719804925961e4bcd17c18596 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:48.105559 19384 tablet_bootstrap.cc:492] T 269eb0e719804925961e4bcd17c18596 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap complete.
I20240502 21:28:48.105775 19384 ts_tablet_manager.cc:1367] T 269eb0e719804925961e4bcd17c18596 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.004s
I20240502 21:28:48.106504 19384 raft_consensus.cc:348] T 269eb0e719804925961e4bcd17c18596 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:28:48.106703 19384 raft_consensus.cc:729] T 269eb0e719804925961e4bcd17c18596 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:28:48.106947 19384 consensus_queue.cc:260] T 269eb0e719804925961e4bcd17c18596 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:28:48.107663 19384 ts_tablet_manager.cc:1398] T 269eb0e719804925961e4bcd17c18596 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:28:48.109786 7505 tablet_copy_service.cc:342] P 31a72286397c4edc9a63eca3412be168: Request end of tablet copy session 71e1375b799b439fa3c83ca763c039c1-269eb0e719804925961e4bcd17c18596 received from {username='slave'} at 127.29.82.65:39791
I20240502 21:28:48.113006 7505 tablet_copy_service.cc:433] P 31a72286397c4edc9a63eca3412be168: ending tablet copy session 71e1375b799b439fa3c83ca763c039c1-269eb0e719804925961e4bcd17c18596 on tablet 269eb0e719804925961e4bcd17c18596 with peer 71e1375b799b439fa3c83ca763c039c1
I20240502 21:28:48.117563 4839 consensus_queue.cc:237] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
W20240502 21:28:48.120290 4770 consensus_peers.cc:485] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): Couldn't send request to peer 31a72286397c4edc9a63eca3412be168. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb81ce7407ab4c9e919a8a5f1b98d545. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:48.121001 4574 raft_consensus.cc:1261] T eb81ce7407ab4c9e919a8a5f1b98d545 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:28:48.121071 4705 raft_consensus.cc:1261] T eb81ce7407ab4c9e919a8a5f1b98d545 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:28:48.121630 18727 consensus_queue.cc:1035] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:48.121904 19122 consensus_queue.cc:1035] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:48.138201 19122 raft_consensus.cc:2932] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:48.138765 4574 raft_consensus.cc:2932] T eb81ce7407ab4c9e919a8a5f1b98d545 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:48.143390 4706 raft_consensus.cc:2932] T eb81ce7407ab4c9e919a8a5f1b98d545 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:48.146678 4297 catalog_manager.cc:5484] T eb81ce7407ab4c9e919a8a5f1b98d545 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: config changed from index -1 to 4, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:48.198766 19413 ts_tablet_manager.cc:902] T 26a00b5700a54672b7389e87c2a4dfbd P 398c3c8f3195492c900439ad5e1931bb: Initiating tablet copy from peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:28:48.202047 19413 tablet_copy_client.cc:308] T 26a00b5700a54672b7389e87c2a4dfbd P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.70:40719
I20240502 21:28:48.202692 7370 tablet_copy_service.cc:140] P d0456c5e55a542128688904e003d403c: Received BeginTabletCopySession request for tablet 26a00b5700a54672b7389e87c2a4dfbd from peer 398c3c8f3195492c900439ad5e1931bb ({username='slave'} at 127.29.82.67:52175)
I20240502 21:28:48.202879 7370 tablet_copy_service.cc:161] P d0456c5e55a542128688904e003d403c: Beginning new tablet copy session on tablet 26a00b5700a54672b7389e87c2a4dfbd from peer 398c3c8f3195492c900439ad5e1931bb at {username='slave'} at 127.29.82.67:52175: session id = 398c3c8f3195492c900439ad5e1931bb-26a00b5700a54672b7389e87c2a4dfbd
I20240502 21:28:48.205734 7370 tablet_copy_source_session.cc:205] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:48.207944 19413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26a00b5700a54672b7389e87c2a4dfbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:48.214654 19413 tablet_copy_client.cc:783] T 26a00b5700a54672b7389e87c2a4dfbd P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:48.214943 19413 tablet_copy_client.cc:647] T 26a00b5700a54672b7389e87c2a4dfbd P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:48.216401 19413 tablet_copy_client.cc:523] T 26a00b5700a54672b7389e87c2a4dfbd P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:48.218760 19413 tablet_bootstrap.cc:492] T 26a00b5700a54672b7389e87c2a4dfbd P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:28:48.225291 19413 tablet_bootstrap.cc:492] T 26a00b5700a54672b7389e87c2a4dfbd P 398c3c8f3195492c900439ad5e1931bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:48.225649 19413 tablet_bootstrap.cc:492] T 26a00b5700a54672b7389e87c2a4dfbd P 398c3c8f3195492c900439ad5e1931bb: Bootstrap complete.
I20240502 21:28:48.225946 19413 ts_tablet_manager.cc:1367] T 26a00b5700a54672b7389e87c2a4dfbd P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240502 21:28:48.226630 19413 raft_consensus.cc:348] T 26a00b5700a54672b7389e87c2a4dfbd P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:48.226810 19413 raft_consensus.cc:729] T 26a00b5700a54672b7389e87c2a4dfbd P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:28:48.227018 19413 consensus_queue.cc:260] T 26a00b5700a54672b7389e87c2a4dfbd P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:48.227694 19413 ts_tablet_manager.cc:1398] T 26a00b5700a54672b7389e87c2a4dfbd P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 21:28:48.228632 7370 tablet_copy_service.cc:342] P d0456c5e55a542128688904e003d403c: Request end of tablet copy session 398c3c8f3195492c900439ad5e1931bb-26a00b5700a54672b7389e87c2a4dfbd received from {username='slave'} at 127.29.82.67:52175
I20240502 21:28:48.228778 7370 tablet_copy_service.cc:433] P d0456c5e55a542128688904e003d403c: ending tablet copy session 398c3c8f3195492c900439ad5e1931bb-26a00b5700a54672b7389e87c2a4dfbd on tablet 26a00b5700a54672b7389e87c2a4dfbd with peer 398c3c8f3195492c900439ad5e1931bb
I20240502 21:28:48.320520 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling FlushMRSOp(a7a0167853fa420586455b82fb893c46): perf score=0.033344
I20240502 21:28:48.321099 4490 maintenance_manager.cc:419] P 71e1375b799b439fa3c83ca763c039c1: Scheduling FlushMRSOp(a7a0167853fa420586455b82fb893c46): perf score=0.033336
I20240502 21:28:48.381435 19446 ts_tablet_manager.cc:902] T 5c5cd4328db94367b4201a49d521a145 P 31a72286397c4edc9a63eca3412be168: Initiating tablet copy from peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
W20240502 21:28:48.382215 4377 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.70:40719, user_credentials={real_user=slave}} blocked reactor thread for 74956.2us
I20240502 21:28:48.456305 19461 ts_tablet_manager.cc:902] T e6f70f1bafd44613abb202a37d4c8b64 P d0456c5e55a542128688904e003d403c: Initiating tablet copy from peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:28:48.465783 19446 tablet_copy_client.cc:308] T 5c5cd4328db94367b4201a49d521a145 P 31a72286397c4edc9a63eca3412be168: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.66:35781
I20240502 21:28:48.466480 4595 tablet_copy_service.cc:140] P 7a736a8ebb164ce0a051c0fccd76c3b8: Received BeginTabletCopySession request for tablet 5c5cd4328db94367b4201a49d521a145 from peer 31a72286397c4edc9a63eca3412be168 ({username='slave'} at 127.29.82.71:55669)
I20240502 21:28:48.466635 4595 tablet_copy_service.cc:161] P 7a736a8ebb164ce0a051c0fccd76c3b8: Beginning new tablet copy session on tablet 5c5cd4328db94367b4201a49d521a145 from peer 31a72286397c4edc9a63eca3412be168 at {username='slave'} at 127.29.82.71:55669: session id = 31a72286397c4edc9a63eca3412be168-5c5cd4328db94367b4201a49d521a145
I20240502 21:28:48.469751 4595 tablet_copy_source_session.cc:205] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:28:48.475772 19446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c5cd4328db94367b4201a49d521a145. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:48.497188 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: FlushMRSOp(a7a0167853fa420586455b82fb893c46) complete. Timing: real 0.176s user 0.043s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":18727,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":8296,"drs_written":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4,"lbm_write_time_us":964,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":34480,"threads_started":2,"wal-append.queue_time_us":13682}
I20240502 21:28:48.500504 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling UndoDeltaBlockGCOp(a7a0167853fa420586455b82fb893c46): 4586 bytes on disk
I20240502 21:28:48.502748 4440 raft_consensus.cc:1203] T 269eb0e719804925961e4bcd17c18596 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:48.503665 19446 tablet_copy_client.cc:783] T 5c5cd4328db94367b4201a49d521a145 P 31a72286397c4edc9a63eca3412be168: tablet copy: Starting download of 5 data blocks...
I20240502 21:28:48.505966 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: UndoDeltaBlockGCOp(a7a0167853fa420586455b82fb893c46) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
I20240502 21:28:48.508484 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling FlushMRSOp(1d0fcf18fd154a72a3bb2f0744febfc7): perf score=0.033390
I20240502 21:28:48.511029 19461 tablet_copy_client.cc:308] T e6f70f1bafd44613abb202a37d4c8b64 P d0456c5e55a542128688904e003d403c: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.69:33193
I20240502 21:28:48.511806 7236 tablet_copy_service.cc:140] P 39981a8dfe94462b9a73ba489af4c714: Received BeginTabletCopySession request for tablet e6f70f1bafd44613abb202a37d4c8b64 from peer d0456c5e55a542128688904e003d403c ({username='slave'} at 127.29.82.70:60977)
I20240502 21:28:48.511966 7236 tablet_copy_service.cc:161] P 39981a8dfe94462b9a73ba489af4c714: Beginning new tablet copy session on tablet e6f70f1bafd44613abb202a37d4c8b64 from peer d0456c5e55a542128688904e003d403c at {username='slave'} at 127.29.82.70:60977: session id = d0456c5e55a542128688904e003d403c-e6f70f1bafd44613abb202a37d4c8b64
I20240502 21:28:48.514781 7236 tablet_copy_source_session.cc:205] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:48.517340 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling FlushMRSOp(1d0fcf18fd154a72a3bb2f0744febfc7): perf score=0.033400
I20240502 21:28:48.523595 19461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6f70f1bafd44613abb202a37d4c8b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:48.530566 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling FlushMRSOp(e5ab4fa95c5a46e9a3377a3c96b9e740): perf score=0.033398
I20240502 21:28:48.546170 19446 tablet_copy_client.cc:647] T 5c5cd4328db94367b4201a49d521a145 P 31a72286397c4edc9a63eca3412be168: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:48.552018 19446 tablet_copy_client.cc:523] T 5c5cd4328db94367b4201a49d521a145 P 31a72286397c4edc9a63eca3412be168: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:48.565614 19461 tablet_copy_client.cc:783] T e6f70f1bafd44613abb202a37d4c8b64 P d0456c5e55a542128688904e003d403c: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:48.565919 19461 tablet_copy_client.cc:647] T e6f70f1bafd44613abb202a37d4c8b64 P d0456c5e55a542128688904e003d403c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:48.568603 19461 tablet_copy_client.cc:523] T e6f70f1bafd44613abb202a37d4c8b64 P d0456c5e55a542128688904e003d403c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:48.575973 19446 tablet_bootstrap.cc:492] T 5c5cd4328db94367b4201a49d521a145 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:28:48.585558 19461 tablet_bootstrap.cc:492] T e6f70f1bafd44613abb202a37d4c8b64 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:28:48.588088 19472 ts_tablet_manager.cc:902] T a33fd7d5a8db41a39ba780e16a84f9b1 P 39981a8dfe94462b9a73ba489af4c714: Initiating tablet copy from peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:28:48.610219 19461 tablet_bootstrap.cc:492] T e6f70f1bafd44613abb202a37d4c8b64 P d0456c5e55a542128688904e003d403c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:48.612987 19461 tablet_bootstrap.cc:492] T e6f70f1bafd44613abb202a37d4c8b64 P d0456c5e55a542128688904e003d403c: Bootstrap complete.
I20240502 21:28:48.613330 19461 ts_tablet_manager.cc:1367] T e6f70f1bafd44613abb202a37d4c8b64 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.003s
I20240502 21:28:48.616405 19461 raft_consensus.cc:348] T e6f70f1bafd44613abb202a37d4c8b64 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:28:48.616696 19461 raft_consensus.cc:729] T e6f70f1bafd44613abb202a37d4c8b64 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:28:48.616950 19461 consensus_queue.cc:260] T e6f70f1bafd44613abb202a37d4c8b64 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:28:48.619414 19461 ts_tablet_manager.cc:1398] T e6f70f1bafd44613abb202a37d4c8b64 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 21:28:48.620072 7236 tablet_copy_service.cc:342] P 39981a8dfe94462b9a73ba489af4c714: Request end of tablet copy session d0456c5e55a542128688904e003d403c-e6f70f1bafd44613abb202a37d4c8b64 received from {username='slave'} at 127.29.82.70:60977
I20240502 21:28:48.620208 7236 tablet_copy_service.cc:433] P 39981a8dfe94462b9a73ba489af4c714: ending tablet copy session d0456c5e55a542128688904e003d403c-e6f70f1bafd44613abb202a37d4c8b64 on tablet e6f70f1bafd44613abb202a37d4c8b64 with peer d0456c5e55a542128688904e003d403c
I20240502 21:28:48.641712 19446 tablet_bootstrap.cc:492] T 5c5cd4328db94367b4201a49d521a145 P 31a72286397c4edc9a63eca3412be168: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:48.642264 19446 tablet_bootstrap.cc:492] T 5c5cd4328db94367b4201a49d521a145 P 31a72286397c4edc9a63eca3412be168: Bootstrap complete.
I20240502 21:28:48.642655 19446 ts_tablet_manager.cc:1367] T 5c5cd4328db94367b4201a49d521a145 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.067s user 0.022s sys 0.000s
I20240502 21:28:48.644241 19446 raft_consensus.cc:348] T 5c5cd4328db94367b4201a49d521a145 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:28:48.644554 19446 raft_consensus.cc:729] T 5c5cd4328db94367b4201a49d521a145 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:28:48.644842 19446 consensus_queue.cc:260] T 5c5cd4328db94367b4201a49d521a145 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:28:48.650449 19446 ts_tablet_manager.cc:1398] T 5c5cd4328db94367b4201a49d521a145 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 21:28:48.651149 4595 tablet_copy_service.cc:342] P 7a736a8ebb164ce0a051c0fccd76c3b8: Request end of tablet copy session 31a72286397c4edc9a63eca3412be168-5c5cd4328db94367b4201a49d521a145 received from {username='slave'} at 127.29.82.71:55669
I20240502 21:28:48.651285 4595 tablet_copy_service.cc:433] P 7a736a8ebb164ce0a051c0fccd76c3b8: ending tablet copy session 31a72286397c4edc9a63eca3412be168-5c5cd4328db94367b4201a49d521a145 on tablet 5c5cd4328db94367b4201a49d521a145 with peer 31a72286397c4edc9a63eca3412be168
I20240502 21:28:48.659680 4706 raft_consensus.cc:1203] T 26a00b5700a54672b7389e87c2a4dfbd P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:48.671564 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:48.690735 19446 ts_tablet_manager.cc:902] T eb81ce7407ab4c9e919a8a5f1b98d545 P 31a72286397c4edc9a63eca3412be168: Initiating tablet copy from peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:28:48.714754 7531 maintenance_manager.cc:419] P 31a72286397c4edc9a63eca3412be168: Scheduling UndoDeltaBlockGCOp(5c5cd4328db94367b4201a49d521a145): 4925 bytes on disk
I20240502 21:28:48.716610 7421 maintenance_manager.cc:643] P 31a72286397c4edc9a63eca3412be168: UndoDeltaBlockGCOp(5c5cd4328db94367b4201a49d521a145) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240502 21:28:48.720737 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:48.738689 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: FlushMRSOp(e5ab4fa95c5a46e9a3377a3c96b9e740) complete. Timing: real 0.208s user 0.044s sys 0.014s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":65580,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":6523,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":999,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":118266,"threads_started":2,"wal-append.queue_time_us":1927}
I20240502 21:28:48.743331 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling UndoDeltaBlockGCOp(e5ab4fa95c5a46e9a3377a3c96b9e740): 4925 bytes on disk
I20240502 21:28:48.744962 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: UndoDeltaBlockGCOp(e5ab4fa95c5a46e9a3377a3c96b9e740) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":135,"lbm_reads_lt_1ms":4}
W20240502 21:28:48.807919 7420 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.67:34537, user_credentials={real_user=slave}} blocked reactor thread for 109470us
I20240502 21:28:48.823801 19472 tablet_copy_client.cc:308] T a33fd7d5a8db41a39ba780e16a84f9b1 P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.66:35781
I20240502 21:28:48.837877 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:48.842084 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:48.842905 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: FlushMRSOp(1d0fcf18fd154a72a3bb2f0744febfc7) complete. Timing: real 0.334s user 0.035s sys 0.021s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":3061,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1229,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4284,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":676,"thread_start_us":49492,"threads_started":1,"wal-append.queue_time_us":43049}
I20240502 21:28:48.843736 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:48.846122 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling UndoDeltaBlockGCOp(1d0fcf18fd154a72a3bb2f0744febfc7): 4840 bytes on disk
I20240502 21:28:48.870124 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: UndoDeltaBlockGCOp(1d0fcf18fd154a72a3bb2f0744febfc7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
I20240502 21:28:48.899235 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: FlushMRSOp(1d0fcf18fd154a72a3bb2f0744febfc7) complete. Timing: real 0.381s user 0.050s sys 0.022s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":172747,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1736,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":32240,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":179389,"threads_started":2,"wal-append.queue_time_us":1227}
I20240502 21:28:48.900326 19446 tablet_copy_client.cc:308] T eb81ce7407ab4c9e919a8a5f1b98d545 P 31a72286397c4edc9a63eca3412be168: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.68:42449
I20240502 21:28:48.901394 4859 tablet_copy_service.cc:140] P 73a1f2ca1f544de9b0e3b37015f85328: Received BeginTabletCopySession request for tablet eb81ce7407ab4c9e919a8a5f1b98d545 from peer 31a72286397c4edc9a63eca3412be168 ({username='slave'} at 127.29.82.71:36305)
I20240502 21:28:48.901571 4859 tablet_copy_service.cc:161] P 73a1f2ca1f544de9b0e3b37015f85328: Beginning new tablet copy session on tablet eb81ce7407ab4c9e919a8a5f1b98d545 from peer 31a72286397c4edc9a63eca3412be168 at {username='slave'} at 127.29.82.71:36305: session id = 31a72286397c4edc9a63eca3412be168-eb81ce7407ab4c9e919a8a5f1b98d545
I20240502 21:28:48.902318 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:48.905265 4859 tablet_copy_source_session.cc:205] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:28:48.908510 19446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb81ce7407ab4c9e919a8a5f1b98d545. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:48.911769 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling FlushMRSOp(e5ab4fa95c5a46e9a3377a3c96b9e740): perf score=0.033506
I20240502 21:28:48.919417 19446 tablet_copy_client.cc:783] T eb81ce7407ab4c9e919a8a5f1b98d545 P 31a72286397c4edc9a63eca3412be168: tablet copy: Starting download of 5 data blocks...
I20240502 21:28:48.922394 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:48.933018 19446 tablet_copy_client.cc:647] T eb81ce7407ab4c9e919a8a5f1b98d545 P 31a72286397c4edc9a63eca3412be168: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:48.934836 19446 tablet_copy_client.cc:523] T eb81ce7407ab4c9e919a8a5f1b98d545 P 31a72286397c4edc9a63eca3412be168: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:48.938902 19446 tablet_bootstrap.cc:492] T eb81ce7407ab4c9e919a8a5f1b98d545 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:28:48.912935 4379 maintenance_manager.cc:643] P 71e1375b799b439fa3c83ca763c039c1: FlushMRSOp(a7a0167853fa420586455b82fb893c46) complete. Timing: real 0.587s user 0.065s sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":410,"dirs.run_cpu_time_us":151,"dirs.run_wall_time_us":1241,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1071,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":63151,"threads_started":2,"wal-append.queue_time_us":62709}
I20240502 21:28:48.943554 4490 maintenance_manager.cc:419] P 71e1375b799b439fa3c83ca763c039c1: Scheduling UndoDeltaBlockGCOp(a7a0167853fa420586455b82fb893c46): 4586 bytes on disk
I20240502 21:28:48.947903 4379 maintenance_manager.cc:643] P 71e1375b799b439fa3c83ca763c039c1: UndoDeltaBlockGCOp(a7a0167853fa420586455b82fb893c46) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240502 21:28:48.951081 4490 maintenance_manager.cc:419] P 71e1375b799b439fa3c83ca763c039c1: Scheduling FlushMRSOp(e5ab4fa95c5a46e9a3377a3c96b9e740): perf score=0.033506
I20240502 21:28:48.953925 7483 raft_consensus.cc:1203] T 5c5cd4328db94367b4201a49d521a145 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:28:48.964958 7347 raft_consensus.cc:1203] T e6f70f1bafd44613abb202a37d4c8b64 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:48.985352 4595 tablet_copy_service.cc:140] P 7a736a8ebb164ce0a051c0fccd76c3b8: Received BeginTabletCopySession request for tablet a33fd7d5a8db41a39ba780e16a84f9b1 from peer 39981a8dfe94462b9a73ba489af4c714 ({username='slave'} at 127.29.82.69:37791)
I20240502 21:28:48.985584 4595 tablet_copy_service.cc:161] P 7a736a8ebb164ce0a051c0fccd76c3b8: Beginning new tablet copy session on tablet a33fd7d5a8db41a39ba780e16a84f9b1 from peer 39981a8dfe94462b9a73ba489af4c714 at {username='slave'} at 127.29.82.69:37791: session id = 39981a8dfe94462b9a73ba489af4c714-a33fd7d5a8db41a39ba780e16a84f9b1
I20240502 21:28:48.990751 4595 tablet_copy_source_session.cc:205] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:28:48.994554 19472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a33fd7d5a8db41a39ba780e16a84f9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:49.002446 19472 tablet_copy_client.cc:783] T a33fd7d5a8db41a39ba780e16a84f9b1 P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Starting download of 5 data blocks...
I20240502 21:28:49.006065 19446 tablet_bootstrap.cc:492] T eb81ce7407ab4c9e919a8a5f1b98d545 P 31a72286397c4edc9a63eca3412be168: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:49.006533 19446 tablet_bootstrap.cc:492] T eb81ce7407ab4c9e919a8a5f1b98d545 P 31a72286397c4edc9a63eca3412be168: Bootstrap complete.
I20240502 21:28:49.006821 19446 ts_tablet_manager.cc:1367] T eb81ce7407ab4c9e919a8a5f1b98d545 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.068s user 0.016s sys 0.001s
I20240502 21:28:49.021790 19446 raft_consensus.cc:348] T eb81ce7407ab4c9e919a8a5f1b98d545 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:28:49.022094 19446 raft_consensus.cc:729] T eb81ce7407ab4c9e919a8a5f1b98d545 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:28:49.030104 19446 consensus_queue.cc:260] T eb81ce7407ab4c9e919a8a5f1b98d545 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:28:49.031719 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:49.033521 19446 ts_tablet_manager.cc:1398] T eb81ce7407ab4c9e919a8a5f1b98d545 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240502 21:28:49.034971 4859 tablet_copy_service.cc:342] P 73a1f2ca1f544de9b0e3b37015f85328: Request end of tablet copy session 31a72286397c4edc9a63eca3412be168-eb81ce7407ab4c9e919a8a5f1b98d545 received from {username='slave'} at 127.29.82.71:36305
I20240502 21:28:49.035084 4859 tablet_copy_service.cc:433] P 73a1f2ca1f544de9b0e3b37015f85328: ending tablet copy session 31a72286397c4edc9a63eca3412be168-eb81ce7407ab4c9e919a8a5f1b98d545 on tablet eb81ce7407ab4c9e919a8a5f1b98d545 with peer 31a72286397c4edc9a63eca3412be168
I20240502 21:28:49.067360 7483 raft_consensus.cc:1203] T eb81ce7407ab4c9e919a8a5f1b98d545 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:28:49.084591 19472 tablet_copy_client.cc:647] T a33fd7d5a8db41a39ba780e16a84f9b1 P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:49.091084 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: FlushMRSOp(e5ab4fa95c5a46e9a3377a3c96b9e740) complete. Timing: real 0.176s user 0.054s sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1519,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":840,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":7240,"threads_started":1,"wal-append.queue_time_us":7112}
I20240502 21:28:49.095973 19472 tablet_copy_client.cc:523] T a33fd7d5a8db41a39ba780e16a84f9b1 P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:49.100339 19472 tablet_bootstrap.cc:492] T a33fd7d5a8db41a39ba780e16a84f9b1 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:28:49.108765 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling UndoDeltaBlockGCOp(e5ab4fa95c5a46e9a3377a3c96b9e740): 4925 bytes on disk
I20240502 21:28:49.109539 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: UndoDeltaBlockGCOp(e5ab4fa95c5a46e9a3377a3c96b9e740) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240502 21:28:49.118883 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling UndoDeltaBlockGCOp(1d0fcf18fd154a72a3bb2f0744febfc7): 4840 bytes on disk
I20240502 21:28:49.122437 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: UndoDeltaBlockGCOp(1d0fcf18fd154a72a3bb2f0744febfc7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240502 21:28:49.141359 4379 maintenance_manager.cc:643] P 71e1375b799b439fa3c83ca763c039c1: FlushMRSOp(e5ab4fa95c5a46e9a3377a3c96b9e740) complete. Timing: real 0.190s user 0.046s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":1166,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":2393,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":869,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":10811,"threads_started":1,"wal-append.queue_time_us":10808}
I20240502 21:28:49.144425 4490 maintenance_manager.cc:419] P 71e1375b799b439fa3c83ca763c039c1: Scheduling UndoDeltaBlockGCOp(e5ab4fa95c5a46e9a3377a3c96b9e740): 4925 bytes on disk
I20240502 21:28:49.145227 4379 maintenance_manager.cc:643] P 71e1375b799b439fa3c83ca763c039c1: UndoDeltaBlockGCOp(e5ab4fa95c5a46e9a3377a3c96b9e740) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240502 21:28:49.224326 7531 maintenance_manager.cc:419] P 31a72286397c4edc9a63eca3412be168: Scheduling UndoDeltaBlockGCOp(eb81ce7407ab4c9e919a8a5f1b98d545): 4925 bytes on disk
I20240502 21:28:49.225029 7421 maintenance_manager.cc:643] P 31a72286397c4edc9a63eca3412be168: UndoDeltaBlockGCOp(eb81ce7407ab4c9e919a8a5f1b98d545) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240502 21:28:49.242163 19472 tablet_bootstrap.cc:492] T a33fd7d5a8db41a39ba780e16a84f9b1 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:49.242683 19472 tablet_bootstrap.cc:492] T a33fd7d5a8db41a39ba780e16a84f9b1 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap complete.
I20240502 21:28:49.243017 19472 ts_tablet_manager.cc:1367] T a33fd7d5a8db41a39ba780e16a84f9b1 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.143s user 0.011s sys 0.012s
I20240502 21:28:49.248977 19472 raft_consensus.cc:348] T a33fd7d5a8db41a39ba780e16a84f9b1 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:28:49.249316 19472 raft_consensus.cc:729] T a33fd7d5a8db41a39ba780e16a84f9b1 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:28:49.249581 19472 consensus_queue.cc:260] T a33fd7d5a8db41a39ba780e16a84f9b1 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:28:49.250272 19472 ts_tablet_manager.cc:1398] T a33fd7d5a8db41a39ba780e16a84f9b1 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240502 21:28:49.251118 4595 tablet_copy_service.cc:342] P 7a736a8ebb164ce0a051c0fccd76c3b8: Request end of tablet copy session 39981a8dfe94462b9a73ba489af4c714-a33fd7d5a8db41a39ba780e16a84f9b1 received from {username='slave'} at 127.29.82.69:37791
I20240502 21:28:49.251222 4595 tablet_copy_service.cc:433] P 7a736a8ebb164ce0a051c0fccd76c3b8: ending tablet copy session 39981a8dfe94462b9a73ba489af4c714-a33fd7d5a8db41a39ba780e16a84f9b1 on tablet a33fd7d5a8db41a39ba780e16a84f9b1 with peer 39981a8dfe94462b9a73ba489af4c714
I20240502 21:28:49.329041 7262 maintenance_manager.cc:419] P 39981a8dfe94462b9a73ba489af4c714: Scheduling UndoDeltaBlockGCOp(a33fd7d5a8db41a39ba780e16a84f9b1): 4840 bytes on disk
I20240502 21:28:49.330705 7152 maintenance_manager.cc:643] P 39981a8dfe94462b9a73ba489af4c714: UndoDeltaBlockGCOp(a33fd7d5a8db41a39ba780e16a84f9b1) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240502 21:28:49.653883 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:49.694530 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:49.699085 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:49.724578 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:49.724645 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:49.731194 7213 raft_consensus.cc:1203] T a33fd7d5a8db41a39ba780e16a84f9b1 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:28:49.761824 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:49.798871 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:49.873124 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:50.014937 4573 consensus_queue.cc:237] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:28:50.017845 4836 raft_consensus.cc:1261] T 5c5cd4328db94367b4201a49d521a145 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:50.018169 7483 raft_consensus.cc:1261] T 5c5cd4328db94367b4201a49d521a145 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:50.018965 19118 consensus_queue.cc:1035] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:50.019315 19537 consensus_queue.cc:1035] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:50.024262 19118 raft_consensus.cc:2932] T 5c5cd4328db94367b4201a49d521a145 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:50.024653 7483 raft_consensus.cc:2932] T 5c5cd4328db94367b4201a49d521a145 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:50.029043 4836 raft_consensus.cc:2932] T 5c5cd4328db94367b4201a49d521a145 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:50.030412 4300 catalog_manager.cc:5484] T 5c5cd4328db94367b4201a49d521a145 P 31a72286397c4edc9a63eca3412be168 reported cstate change: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:50.074142 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c5cd4328db94367b4201a49d521a145 with delete_type TABLET_DATA_TOMBSTONED (TS 398c3c8f3195492c900439ad5e1931bb not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44764
I20240502 21:28:50.074793 19626 tablet_replica.cc:330] T 5c5cd4328db94367b4201a49d521a145 P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:28:50.076028 19626 raft_consensus.cc:2229] T 5c5cd4328db94367b4201a49d521a145 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:50.076464 19626 raft_consensus.cc:2258] T 5c5cd4328db94367b4201a49d521a145 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:50.077585 19626 ts_tablet_manager.cc:1875] T 5c5cd4328db94367b4201a49d521a145 P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:50.113492 19626 ts_tablet_manager.cc:1888] T 5c5cd4328db94367b4201a49d521a145 P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:28:50.113900 19626 log.cc:1196] T 5c5cd4328db94367b4201a49d521a145 P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/5c5cd4328db94367b4201a49d521a145
I20240502 21:28:50.115754 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet 5c5cd4328db94367b4201a49d521a145 (table rebalance_test_table_13 [id=d14fe34dcad44629af56956cc9e71e5b]) successfully deleted
I20240502 21:28:50.342099 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:50.381731 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:50.417884 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:50.439510 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:50.444871 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:50.484992 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 21:28:50.495496 4773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.67:34537, user_credentials={real_user=slave}} blocked reactor thread for 86256.1us
I20240502 21:28:50.532105 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:50.536717 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:50.713171 7482 consensus_queue.cc:237] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:28:50.715157 7347 raft_consensus.cc:1261] T 269eb0e719804925961e4bcd17c18596 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:50.715157 4441 raft_consensus.cc:1261] T 269eb0e719804925961e4bcd17c18596 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:50.715689 19300 consensus_queue.cc:1035] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:50.716008 19487 consensus_queue.cc:1035] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:50.720397 19300 raft_consensus.cc:2932] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:50.720937 7347 raft_consensus.cc:2932] T 269eb0e719804925961e4bcd17c18596 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:50.722270 4441 raft_consensus.cc:2932] T 269eb0e719804925961e4bcd17c18596 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:50.730247 4297 catalog_manager.cc:5484] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 reported cstate change: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:50.736405 7597 tablet_service.cc:1514] Processing DeleteTablet for tablet 269eb0e719804925961e4bcd17c18596 with delete_type TABLET_DATA_TOMBSTONED (TS 0ddd486109f5434cace13d60d633ac04 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44444
I20240502 21:28:50.745266 19717 tablet_replica.cc:330] T 269eb0e719804925961e4bcd17c18596 P 0ddd486109f5434cace13d60d633ac04: stopping tablet replica
I20240502 21:28:50.745760 19717 raft_consensus.cc:2229] T 269eb0e719804925961e4bcd17c18596 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:50.746186 19717 raft_consensus.cc:2258] T 269eb0e719804925961e4bcd17c18596 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:50.746846 19717 ts_tablet_manager.cc:1875] T 269eb0e719804925961e4bcd17c18596 P 0ddd486109f5434cace13d60d633ac04: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:50.753414 19717 ts_tablet_manager.cc:1888] T 269eb0e719804925961e4bcd17c18596 P 0ddd486109f5434cace13d60d633ac04: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:50.753515 19717 log.cc:1196] T 269eb0e719804925961e4bcd17c18596 P 0ddd486109f5434cace13d60d633ac04: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/wals/269eb0e719804925961e4bcd17c18596
I20240502 21:28:50.754182 4285 catalog_manager.cc:4830] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): tablet 269eb0e719804925961e4bcd17c18596 (table rebalancer_extra_table_32 [id=9606abae682a4594ada645db2e56a7d4]) successfully deleted
W20240502 21:28:51.098842 7551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.70:40719, user_credentials={real_user=slave}} blocked reactor thread for 88016.8us
I20240502 21:28:51.264401 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:51.303277 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:51.307682 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:51.309720 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:51.380847 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:51.382661 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:51.403977 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:51.468966 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:51.602536 4705 consensus_queue.cc:237] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:28:51.604585 4837 raft_consensus.cc:1261] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:51.604621 7213 raft_consensus.cc:1261] T 5f614ef09d764740926c10618370084f P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:51.605127 18476 consensus_queue.cc:1035] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:51.605708 18965 consensus_queue.cc:1035] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:51.609114 18476 raft_consensus.cc:2932] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:51.609606 4837 raft_consensus.cc:2932] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:51.616988 4292 catalog_manager.cc:5484] T 5f614ef09d764740926c10618370084f P 398c3c8f3195492c900439ad5e1931bb reported cstate change: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:51.619990 7213 raft_consensus.cc:2932] T 5f614ef09d764740926c10618370084f P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:51.633097 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f614ef09d764740926c10618370084f with delete_type TABLET_DATA_TOMBSTONED (TS 7a736a8ebb164ce0a051c0fccd76c3b8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60018
I20240502 21:28:51.633773 19824 tablet_replica.cc:330] T 5f614ef09d764740926c10618370084f P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:28:51.634220 19824 raft_consensus.cc:2229] T 5f614ef09d764740926c10618370084f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:51.634616 19824 raft_consensus.cc:2258] T 5f614ef09d764740926c10618370084f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:51.636747 19824 ts_tablet_manager.cc:1875] T 5f614ef09d764740926c10618370084f P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:51.655925 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling FlushMRSOp(104c676bff964b8b8566b2200ce238c7): perf score=0.033351
I20240502 21:28:51.673270 19824 ts_tablet_manager.cc:1888] T 5f614ef09d764740926c10618370084f P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:28:51.673570 19824 log.cc:1196] T 5f614ef09d764740926c10618370084f P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/5f614ef09d764740926c10618370084f
I20240502 21:28:51.674434 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet 5f614ef09d764740926c10618370084f (table rebalance_test_table_9 [id=80b493b0fc3f49dcbf948cee9e748e3c]) successfully deleted
I20240502 21:28:51.676541 4490 maintenance_manager.cc:419] P 71e1375b799b439fa3c83ca763c039c1: Scheduling FlushMRSOp(1c243b27429043aa8642d9b2ed595ab3): perf score=0.033353
I20240502 21:28:51.678438 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling FlushMRSOp(4dc27c0c9cee447f84e4bc1a63f64b68): perf score=0.033347
I20240502 21:28:51.754933 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: FlushMRSOp(104c676bff964b8b8566b2200ce238c7) complete. Timing: real 0.099s user 0.048s sys 0.014s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":15368,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":2763,"drs_written":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2261,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":48122,"threads_started":2,"wal-append.queue_time_us":35619}
I20240502 21:28:51.759224 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling UndoDeltaBlockGCOp(104c676bff964b8b8566b2200ce238c7): 4925 bytes on disk
I20240502 21:28:51.762457 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: UndoDeltaBlockGCOp(104c676bff964b8b8566b2200ce238c7) complete. Timing: real 0.003s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":151,"lbm_reads_lt_1ms":4}
I20240502 21:28:51.770875 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling FlushMRSOp(4dc27c0c9cee447f84e4bc1a63f64b68): perf score=0.033375
I20240502 21:28:51.788776 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling FlushMRSOp(4dc27c0c9cee447f84e4bc1a63f64b68): perf score=0.033379
I20240502 21:28:51.809192 4379 maintenance_manager.cc:643] P 71e1375b799b439fa3c83ca763c039c1: FlushMRSOp(1c243b27429043aa8642d9b2ed595ab3) complete. Timing: real 0.132s user 0.037s sys 0.026s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":10277,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":9099,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":885,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":17049,"threads_started":2,"wal-append.queue_time_us":6536}
I20240502 21:28:51.819322 4490 maintenance_manager.cc:419] P 71e1375b799b439fa3c83ca763c039c1: Scheduling UndoDeltaBlockGCOp(1c243b27429043aa8642d9b2ed595ab3): 4841 bytes on disk
I20240502 21:28:51.824126 4379 maintenance_manager.cc:643] P 71e1375b799b439fa3c83ca763c039c1: UndoDeltaBlockGCOp(1c243b27429043aa8642d9b2ed595ab3) complete. Timing: real 0.004s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240502 21:28:51.834810 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: FlushMRSOp(4dc27c0c9cee447f84e4bc1a63f64b68) complete. Timing: real 0.156s user 0.065s sys 0.021s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":149,"dirs.run_wall_time_us":2035,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":966,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":55443,"threads_started":1,"wal-append.queue_time_us":55308}
I20240502 21:28:51.841806 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling UndoDeltaBlockGCOp(4dc27c0c9cee447f84e4bc1a63f64b68): 4586 bytes on disk
I20240502 21:28:51.842501 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: UndoDeltaBlockGCOp(4dc27c0c9cee447f84e4bc1a63f64b68) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240502 21:28:51.934290 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: FlushMRSOp(4dc27c0c9cee447f84e4bc1a63f64b68) complete. Timing: real 0.145s user 0.044s sys 0.028s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":17150,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":11103,"drs_written":1,"lbm_read_time_us":1058,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":7451,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":36987,"threads_started":2,"wal-append.queue_time_us":20362}
I20240502 21:28:51.938083 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling UndoDeltaBlockGCOp(4dc27c0c9cee447f84e4bc1a63f64b68): 4586 bytes on disk
I20240502 21:28:51.944152 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: FlushMRSOp(4dc27c0c9cee447f84e4bc1a63f64b68) complete. Timing: real 0.173s user 0.062s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":97,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":2180,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1041,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":5042,"threads_started":1,"wal-append.queue_time_us":1192}
I20240502 21:28:51.943771 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: UndoDeltaBlockGCOp(4dc27c0c9cee447f84e4bc1a63f64b68) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240502 21:28:51.947266 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling UndoDeltaBlockGCOp(4dc27c0c9cee447f84e4bc1a63f64b68): 4586 bytes on disk
I20240502 21:28:51.947378 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling FlushMRSOp(104c676bff964b8b8566b2200ce238c7): perf score=0.033421
I20240502 21:28:51.952231 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: UndoDeltaBlockGCOp(4dc27c0c9cee447f84e4bc1a63f64b68) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240502 21:28:52.010169 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: FlushMRSOp(104c676bff964b8b8566b2200ce238c7) complete. Timing: real 0.062s user 0.047s sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":354,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":1973,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":838,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":1280,"thread_start_us":244,"threads_started":1,"wal-append.queue_time_us":336}
I20240502 21:28:52.028347 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling FlushMRSOp(1c243b27429043aa8642d9b2ed595ab3): perf score=0.033438
I20240502 21:28:52.097702 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: FlushMRSOp(1c243b27429043aa8642d9b2ed595ab3) complete. Timing: real 0.069s user 0.034s sys 0.006s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":1869,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":781,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":23432,"threads_started":1,"wal-append.queue_time_us":23305}
I20240502 21:28:52.100844 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling UndoDeltaBlockGCOp(104c676bff964b8b8566b2200ce238c7): 4925 bytes on disk
I20240502 21:28:52.101466 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: UndoDeltaBlockGCOp(104c676bff964b8b8566b2200ce238c7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240502 21:28:52.105795 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling UndoDeltaBlockGCOp(1c243b27429043aa8642d9b2ed595ab3): 4841 bytes on disk
I20240502 21:28:52.107620 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: UndoDeltaBlockGCOp(1c243b27429043aa8642d9b2ed595ab3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":181,"lbm_reads_lt_1ms":4}
I20240502 21:28:52.307217 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:52.314215 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:52.319801 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:52.381836 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 21:28:52.412258 4642 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.66:35781, user_credentials={real_user=slave}} blocked reactor thread for 75121.3us
I20240502 21:28:52.417529 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:52.430936 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:52.499315 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:52.503849 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:52.647755 7350 consensus_queue.cc:237] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:52.656790 4574 raft_consensus.cc:1261] T 26a00b5700a54672b7389e87c2a4dfbd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:52.657335 19934 consensus_queue.cc:1035] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:52.657537 4705 raft_consensus.cc:1261] T 26a00b5700a54672b7389e87c2a4dfbd P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:52.658567 19934 consensus_queue.cc:1035] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:52.662325 19934 raft_consensus.cc:2932] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:52.666869 4574 raft_consensus.cc:2932] T 26a00b5700a54672b7389e87c2a4dfbd P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:52.672451 4296 catalog_manager.cc:5484] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c reported cstate change: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:52.689414 4705 raft_consensus.cc:2932] T 26a00b5700a54672b7389e87c2a4dfbd P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:52.706715 7194 tablet_service.cc:1514] Processing DeleteTablet for tablet 26a00b5700a54672b7389e87c2a4dfbd with delete_type TABLET_DATA_TOMBSTONED (TS 39981a8dfe94462b9a73ba489af4c714 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44652
I20240502 21:28:52.720885 19952 tablet_replica.cc:330] T 26a00b5700a54672b7389e87c2a4dfbd P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:28:52.721637 19952 raft_consensus.cc:2229] T 26a00b5700a54672b7389e87c2a4dfbd P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:52.721935 19952 raft_consensus.cc:2258] T 26a00b5700a54672b7389e87c2a4dfbd P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:52.722734 19952 ts_tablet_manager.cc:1875] T 26a00b5700a54672b7389e87c2a4dfbd P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:52.728212 19952 ts_tablet_manager.cc:1888] T 26a00b5700a54672b7389e87c2a4dfbd P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:52.728356 19952 log.cc:1196] T 26a00b5700a54672b7389e87c2a4dfbd P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/26a00b5700a54672b7389e87c2a4dfbd
I20240502 21:28:52.729171 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet 26a00b5700a54672b7389e87c2a4dfbd (table rebalancer_extra_table_10 [id=d5866bdd67f64fe5bbeebaed4bd1f67e]) successfully deleted
I20240502 21:28:53.102615 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:53.109393 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:53.112582 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:53.134021 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:53.140612 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:53.170770 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:53.247666 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:53.347250 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:53.547395 7214 consensus_queue.cc:237] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:28:53.549371 7347 raft_consensus.cc:1261] T e6f70f1bafd44613abb202a37d4c8b64 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:53.549679 4837 raft_consensus.cc:1261] T e6f70f1bafd44613abb202a37d4c8b64 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:53.549983 19688 consensus_queue.cc:1035] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:53.550434 19688 consensus_queue.cc:1035] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:53.553503 19688 raft_consensus.cc:2932] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:53.554123 7347 raft_consensus.cc:2932] T e6f70f1bafd44613abb202a37d4c8b64 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:53.561769 4292 catalog_manager.cc:5484] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:53.575090 4837 raft_consensus.cc:2932] T e6f70f1bafd44613abb202a37d4c8b64 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:53.579286 7462 tablet_service.cc:1514] Processing DeleteTablet for tablet e6f70f1bafd44613abb202a37d4c8b64 with delete_type TABLET_DATA_TOMBSTONED (TS 31a72286397c4edc9a63eca3412be168 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:37182
I20240502 21:28:53.580006 20054 tablet_replica.cc:330] T e6f70f1bafd44613abb202a37d4c8b64 P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:28:53.580238 20054 raft_consensus.cc:2229] T e6f70f1bafd44613abb202a37d4c8b64 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:53.580497 20054 raft_consensus.cc:2258] T e6f70f1bafd44613abb202a37d4c8b64 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:53.581209 20054 ts_tablet_manager.cc:1875] T e6f70f1bafd44613abb202a37d4c8b64 P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:53.588348 20054 ts_tablet_manager.cc:1888] T e6f70f1bafd44613abb202a37d4c8b64 P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:53.588464 20054 log.cc:1196] T e6f70f1bafd44613abb202a37d4c8b64 P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/e6f70f1bafd44613abb202a37d4c8b64
I20240502 21:28:53.589241 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet e6f70f1bafd44613abb202a37d4c8b64 (table rebalancer_extra_table_28 [id=37d345f972f34b419de6c03d44f832d8]) successfully deleted
I20240502 21:28:53.967516 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:54.015260 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:54.029587 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:54.128803 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:54.136343 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:54.161209 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:54.185005 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:54.235814 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 21:28:54.248670 4378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.67:34537, user_credentials={real_user=slave}} blocked reactor thread for 81281.1us
I20240502 21:28:54.365576 4490 maintenance_manager.cc:419] P 71e1375b799b439fa3c83ca763c039c1: Scheduling FlushMRSOp(b7c60d990751440392ab6e5feb0d8498): perf score=0.033353
I20240502 21:28:54.385403 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling FlushMRSOp(38ce97f9f9ac4398a16958931ef7262f): perf score=0.033356
I20240502 21:28:54.402029 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling FlushMRSOp(38ce97f9f9ac4398a16958931ef7262f): perf score=0.033374
I20240502 21:28:54.486847 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling FlushMRSOp(b7c60d990751440392ab6e5feb0d8498): perf score=0.033372
I20240502 21:28:54.528790 4379 maintenance_manager.cc:643] P 71e1375b799b439fa3c83ca763c039c1: FlushMRSOp(b7c60d990751440392ab6e5feb0d8498) complete. Timing: real 0.163s user 0.053s sys 0.049s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":25080,"dirs.run_cpu_time_us":262,"dirs.run_wall_time_us":4510,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1521,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":90678,"threads_started":2,"wal-append.queue_time_us":65832}
I20240502 21:28:54.531760 4490 maintenance_manager.cc:419] P 71e1375b799b439fa3c83ca763c039c1: Scheduling UndoDeltaBlockGCOp(b7c60d990751440392ab6e5feb0d8498): 4586 bytes on disk
I20240502 21:28:54.535176 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: FlushMRSOp(38ce97f9f9ac4398a16958931ef7262f) complete. Timing: real 0.133s user 0.054s sys 0.037s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":58845,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":2076,"drs_written":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4410,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":59096,"threads_started":2,"wal-append.queue_time_us":332}
I20240502 21:28:54.535543 4379 maintenance_manager.cc:643] P 71e1375b799b439fa3c83ca763c039c1: UndoDeltaBlockGCOp(b7c60d990751440392ab6e5feb0d8498) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240502 21:28:54.541469 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling FlushMRSOp(3a1fadb11cd6436a94e4a67a3aad17ba): perf score=0.033401
I20240502 21:28:54.573844 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: FlushMRSOp(38ce97f9f9ac4398a16958931ef7262f) complete. Timing: real 0.188s user 0.051s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":20150,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":2410,"drs_written":1,"lbm_read_time_us":7850,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":1183,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":49403,"threads_started":2,"wal-append.queue_time_us":22725}
I20240502 21:28:54.577064 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling FlushMRSOp(3a1fadb11cd6436a94e4a67a3aad17ba): perf score=0.033403
I20240502 21:28:54.611539 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: FlushMRSOp(b7c60d990751440392ab6e5feb0d8498) complete. Timing: real 0.124s user 0.044s sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3545,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":7664,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1021,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":10560,"threads_started":2,"wal-append.queue_time_us":6798}
I20240502 21:28:54.614502 4885 maintenance_manager.cc:419] P 73a1f2ca1f544de9b0e3b37015f85328: Scheduling UndoDeltaBlockGCOp(b7c60d990751440392ab6e5feb0d8498): 4586 bytes on disk
I20240502 21:28:54.616011 4774 maintenance_manager.cc:643] P 73a1f2ca1f544de9b0e3b37015f85328: UndoDeltaBlockGCOp(b7c60d990751440392ab6e5feb0d8498) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240502 21:28:54.622275 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: FlushMRSOp(3a1fadb11cd6436a94e4a67a3aad17ba) complete. Timing: real 0.078s user 0.044s sys 0.005s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":454,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":4513,"drs_written":1,"lbm_read_time_us":128,"lbm_reads_lt_1ms":4,"lbm_write_time_us":955,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":8901,"threads_started":1,"wal-append.queue_time_us":8751}
I20240502 21:28:54.626070 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling UndoDeltaBlockGCOp(38ce97f9f9ac4398a16958931ef7262f): 4925 bytes on disk
I20240502 21:28:54.626768 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: UndoDeltaBlockGCOp(38ce97f9f9ac4398a16958931ef7262f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240502 21:28:54.631870 4622 maintenance_manager.cc:419] P 7a736a8ebb164ce0a051c0fccd76c3b8: Scheduling UndoDeltaBlockGCOp(3a1fadb11cd6436a94e4a67a3aad17ba): 4840 bytes on disk
I20240502 21:28:54.632366 4512 maintenance_manager.cc:643] P 7a736a8ebb164ce0a051c0fccd76c3b8: UndoDeltaBlockGCOp(3a1fadb11cd6436a94e4a67a3aad17ba) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240502 21:28:54.646041 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: FlushMRSOp(3a1fadb11cd6436a94e4a67a3aad17ba) complete. Timing: real 0.068s user 0.034s sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":106,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":1055,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":955,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":6408,"threads_started":1,"wal-append.queue_time_us":6507}
I20240502 21:28:54.649191 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling UndoDeltaBlockGCOp(38ce97f9f9ac4398a16958931ef7262f): 4925 bytes on disk
I20240502 21:28:54.649331 4836 consensus_queue.cc:237] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:28:54.651393 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: UndoDeltaBlockGCOp(38ce97f9f9ac4398a16958931ef7262f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240502 21:28:54.654116 4753 maintenance_manager.cc:419] P 398c3c8f3195492c900439ad5e1931bb: Scheduling UndoDeltaBlockGCOp(3a1fadb11cd6436a94e4a67a3aad17ba): 4840 bytes on disk
I20240502 21:28:54.654814 4643 maintenance_manager.cc:643] P 398c3c8f3195492c900439ad5e1931bb: UndoDeltaBlockGCOp(3a1fadb11cd6436a94e4a67a3aad17ba) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":126,"lbm_reads_lt_1ms":4}
I20240502 21:28:54.657048 7482 raft_consensus.cc:1261] T eb81ce7407ab4c9e919a8a5f1b98d545 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:54.657521 18727 consensus_queue.cc:1035] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:54.658668 4705 raft_consensus.cc:1261] T eb81ce7407ab4c9e919a8a5f1b98d545 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:54.659410 18727 consensus_queue.cc:1035] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:54.660622 18727 raft_consensus.cc:2932] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:54.664772 7482 raft_consensus.cc:2932] T eb81ce7407ab4c9e919a8a5f1b98d545 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:54.669325 4705 raft_consensus.cc:2932] T eb81ce7407ab4c9e919a8a5f1b98d545 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:54.670436 4297 catalog_manager.cc:5484] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:28:54.710419 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet eb81ce7407ab4c9e919a8a5f1b98d545 with delete_type TABLET_DATA_TOMBSTONED (TS 7a736a8ebb164ce0a051c0fccd76c3b8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60018
I20240502 21:28:54.714179 20175 tablet_replica.cc:330] T eb81ce7407ab4c9e919a8a5f1b98d545 P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:28:54.714504 20175 raft_consensus.cc:2229] T eb81ce7407ab4c9e919a8a5f1b98d545 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:54.714777 20175 raft_consensus.cc:2258] T eb81ce7407ab4c9e919a8a5f1b98d545 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:54.715730 20175 ts_tablet_manager.cc:1875] T eb81ce7407ab4c9e919a8a5f1b98d545 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:54.731242 20175 ts_tablet_manager.cc:1888] T eb81ce7407ab4c9e919a8a5f1b98d545 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:28:54.731412 20175 log.cc:1196] T eb81ce7407ab4c9e919a8a5f1b98d545 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/eb81ce7407ab4c9e919a8a5f1b98d545
I20240502 21:28:54.732918 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet eb81ce7407ab4c9e919a8a5f1b98d545 (table rebalance_test_table_1 [id=d6c511862aad4ba5a222ddedb7e95c27]) successfully deleted
I20240502 21:28:54.865721 7483 consensus_queue.cc:237] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:28:54.873526 7214 raft_consensus.cc:1261] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:54.873562 7617 raft_consensus.cc:1261] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:54.874616 20144 consensus_queue.cc:1035] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:54.875247 20144 consensus_queue.cc:1035] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:28:54.880005 7416 consensus_peers.cc:485] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 063bdcf9cea74d5cbe3143eb16fba6a5. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:54.881363 20144 raft_consensus.cc:2932] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:54.881978 7617 raft_consensus.cc:2932] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:54.888430 7213 raft_consensus.cc:2932] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:54.888883 4291 catalog_manager.cc:5484] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 reported cstate change: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:54.941222 4440 consensus_queue.cc:237] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
W20240502 21:28:54.943826 4377 consensus_peers.cc:485] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1 -> Peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Couldn't send request to peer d0456c5e55a542128688904e003d403c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6687a533112445dc86951fc778d0725b. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:54.944209 4705 raft_consensus.cc:1261] T 6687a533112445dc86951fc778d0725b P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:28:54.944257 4574 raft_consensus.cc:1261] T 6687a533112445dc86951fc778d0725b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:28:54.944842 20130 consensus_queue.cc:1035] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:54.945248 20237 consensus_queue.cc:1035] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:54.948247 20237 raft_consensus.cc:2932] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:54.948928 4574 raft_consensus.cc:2932] T 6687a533112445dc86951fc778d0725b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:54.955773 4293 catalog_manager.cc:5484] T 6687a533112445dc86951fc778d0725b P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: config changed from index -1 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:54.957677 4705 raft_consensus.cc:2932] T 6687a533112445dc86951fc778d0725b P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:54.996704 7350 consensus_queue.cc:237] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:28:54.999644 7616 raft_consensus.cc:1261] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:54.999886 7213 raft_consensus.cc:1261] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:55.000484 20258 consensus_queue.cc:1035] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:55.000783 19944 consensus_queue.cc:1035] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:28:55.001041 7281 consensus_peers.cc:485] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d095a5e8f1f34b5ea2b6797ef06f79c6. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:55.003507 20258 raft_consensus.cc:2932] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:55.004099 7213 raft_consensus.cc:2932] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:55.010082 7616 raft_consensus.cc:2932] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:55.011786 4297 catalog_manager.cc:5484] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:55.081532 7483 consensus_queue.cc:237] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:28:55.084045 7616 raft_consensus.cc:1261] T ba5b4c81211647dd8383ec9484bb9c75 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:28:55.084407 7416 consensus_peers.cc:485] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba5b4c81211647dd8383ec9484bb9c75. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:55.085659 19852 consensus_queue.cc:1035] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:55.086374 4705 raft_consensus.cc:1261] T ba5b4c81211647dd8383ec9484bb9c75 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:55.088611 19854 consensus_queue.cc:1035] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:55.092149 19852 raft_consensus.cc:2932] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:55.093298 4705 raft_consensus.cc:2932] T ba5b4c81211647dd8383ec9484bb9c75 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:55.097210 7616 raft_consensus.cc:2932] T ba5b4c81211647dd8383ec9484bb9c75 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:55.098879 4293 catalog_manager.cc:5484] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 reported cstate change: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:55.155539 4836 consensus_queue.cc:237] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:55.157884 7350 raft_consensus.cc:1261] T 686a74b8c1034112b293c4c7c6bcd99a P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:28:55.158035 4773 consensus_peers.cc:485] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 686a74b8c1034112b293c4c7c6bcd99a. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:55.158048 7483 raft_consensus.cc:1261] T 686a74b8c1034112b293c4c7c6bcd99a P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:55.158421 19685 consensus_queue.cc:1035] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:55.158685 18727 consensus_queue.cc:1035] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:55.161235 19685 raft_consensus.cc:2932] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:55.161830 7483 raft_consensus.cc:2932] T 686a74b8c1034112b293c4c7c6bcd99a P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:55.166874 7350 raft_consensus.cc:2932] T 686a74b8c1034112b293c4c7c6bcd99a P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:55.167007 4297 catalog_manager.cc:5484] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:55.209347 4836 consensus_queue.cc:237] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:28:55.212497 7616 raft_consensus.cc:1261] T 0c56940914cb4161bd258bf881f06c07 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:28:55.212939 4770 consensus_peers.cc:485] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0c56940914cb4161bd258bf881f06c07. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:55.213191 7213 raft_consensus.cc:1261] T 0c56940914cb4161bd258bf881f06c07 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:55.213541 20297 consensus_queue.cc:1035] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:55.213827 18727 consensus_queue.cc:1035] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:55.219714 20297 raft_consensus.cc:2932] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:55.220203 7213 raft_consensus.cc:2932] T 0c56940914cb4161bd258bf881f06c07 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:55.223853 7616 raft_consensus.cc:2932] T 0c56940914cb4161bd258bf881f06c07 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:55.227097 4291 catalog_manager.cc:5484] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:55.281093 4836 consensus_queue.cc:237] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:28:55.283131 4440 raft_consensus.cc:1261] T b5099940a48f4466aa7c713bb88d108e P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:28:55.283268 4772 consensus_peers.cc:485] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Couldn't send request to peer 39981a8dfe94462b9a73ba489af4c714. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5099940a48f4466aa7c713bb88d108e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:55.283681 4706 raft_consensus.cc:1261] T b5099940a48f4466aa7c713bb88d108e P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:28:55.283689 19685 consensus_queue.cc:1035] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:55.284212 18727 consensus_queue.cc:1035] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:55.292929 18727 raft_consensus.cc:2932] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:55.296144 4706 raft_consensus.cc:2932] T b5099940a48f4466aa7c713bb88d108e P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:55.300096 4291 catalog_manager.cc:5484] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index -1 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:55.311619 20344 ts_tablet_manager.cc:902] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 73a1f2ca1f544de9b0e3b37015f85328: Initiating tablet copy from peer 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:28:55.312258 4440 raft_consensus.cc:2932] T b5099940a48f4466aa7c713bb88d108e P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:55.324628 20344 tablet_copy_client.cc:308] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.71:33913
I20240502 21:28:55.325354 7505 tablet_copy_service.cc:140] P 31a72286397c4edc9a63eca3412be168: Received BeginTabletCopySession request for tablet 063bdcf9cea74d5cbe3143eb16fba6a5 from peer 73a1f2ca1f544de9b0e3b37015f85328 ({username='slave'} at 127.29.82.68:47963)
I20240502 21:28:55.325531 7505 tablet_copy_service.cc:161] P 31a72286397c4edc9a63eca3412be168: Beginning new tablet copy session on tablet 063bdcf9cea74d5cbe3143eb16fba6a5 from peer 73a1f2ca1f544de9b0e3b37015f85328 at {username='slave'} at 127.29.82.68:47963: session id = 73a1f2ca1f544de9b0e3b37015f85328-063bdcf9cea74d5cbe3143eb16fba6a5
I20240502 21:28:55.328846 7505 tablet_copy_source_session.cc:205] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:55.330698 20344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 063bdcf9cea74d5cbe3143eb16fba6a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:55.343832 20344 tablet_copy_client.cc:783] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:55.344040 20344 tablet_copy_client.cc:647] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:55.345752 20344 tablet_copy_client.cc:523] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:55.348562 20344 tablet_bootstrap.cc:492] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:28:55.363720 20344 tablet_bootstrap.cc:492] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:55.364113 20344 tablet_bootstrap.cc:492] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap complete.
I20240502 21:28:55.364365 20344 ts_tablet_manager.cc:1367] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.009s
I20240502 21:28:55.365103 20344 raft_consensus.cc:348] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:28:55.365350 20344 raft_consensus.cc:729] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:28:55.365664 20344 consensus_queue.cc:260] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:28:55.366514 20344 ts_tablet_manager.cc:1398] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:28:55.367414 7505 tablet_copy_service.cc:342] P 31a72286397c4edc9a63eca3412be168: Request end of tablet copy session 73a1f2ca1f544de9b0e3b37015f85328-063bdcf9cea74d5cbe3143eb16fba6a5 received from {username='slave'} at 127.29.82.68:47963
I20240502 21:28:55.367630 7505 tablet_copy_service.cc:433] P 31a72286397c4edc9a63eca3412be168: ending tablet copy session 73a1f2ca1f544de9b0e3b37015f85328-063bdcf9cea74d5cbe3143eb16fba6a5 on tablet 063bdcf9cea74d5cbe3143eb16fba6a5 with peer 73a1f2ca1f544de9b0e3b37015f85328
I20240502 21:28:55.407532 7617 consensus_queue.cc:237] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:28:55.410610 7482 raft_consensus.cc:1261] T 6065bdba6e2840b8b368d8f92d1fa260 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:28:55.411119 7552 consensus_peers.cc:485] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6065bdba6e2840b8b368d8f92d1fa260. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:55.411386 7213 raft_consensus.cc:1261] T 6065bdba6e2840b8b368d8f92d1fa260 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:55.412102 20112 consensus_queue.cc:1035] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:55.412381 19761 consensus_queue.cc:1035] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:28:55.428624 19761 raft_consensus.cc:2932] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:55.429114 7214 raft_consensus.cc:2932] T 6065bdba6e2840b8b368d8f92d1fa260 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:55.432605 7483 raft_consensus.cc:2932] T 6065bdba6e2840b8b368d8f92d1fa260 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:55.435194 4293 catalog_manager.cc:5484] T 6065bdba6e2840b8b368d8f92d1fa260 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:28:55.460662 20344 ts_tablet_manager.cc:902] T ba5b4c81211647dd8383ec9484bb9c75 P 73a1f2ca1f544de9b0e3b37015f85328: Initiating tablet copy from peer 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:28:55.484387 20344 tablet_copy_client.cc:308] T ba5b4c81211647dd8383ec9484bb9c75 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.71:33913
I20240502 21:28:55.485154 7505 tablet_copy_service.cc:140] P 31a72286397c4edc9a63eca3412be168: Received BeginTabletCopySession request for tablet ba5b4c81211647dd8383ec9484bb9c75 from peer 73a1f2ca1f544de9b0e3b37015f85328 ({username='slave'} at 127.29.82.68:47963)
I20240502 21:28:55.485338 7505 tablet_copy_service.cc:161] P 31a72286397c4edc9a63eca3412be168: Beginning new tablet copy session on tablet ba5b4c81211647dd8383ec9484bb9c75 from peer 73a1f2ca1f544de9b0e3b37015f85328 at {username='slave'} at 127.29.82.68:47963: session id = 73a1f2ca1f544de9b0e3b37015f85328-ba5b4c81211647dd8383ec9484bb9c75
I20240502 21:28:55.490418 7505 tablet_copy_source_session.cc:205] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:55.491897 20344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba5b4c81211647dd8383ec9484bb9c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:55.498556 20344 tablet_copy_client.cc:783] T ba5b4c81211647dd8383ec9484bb9c75 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:55.498836 20344 tablet_copy_client.cc:647] T ba5b4c81211647dd8383ec9484bb9c75 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:55.508893 20344 tablet_copy_client.cc:523] T ba5b4c81211647dd8383ec9484bb9c75 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:55.513278 20344 tablet_bootstrap.cc:492] T ba5b4c81211647dd8383ec9484bb9c75 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:28:55.520107 20344 tablet_bootstrap.cc:492] T ba5b4c81211647dd8383ec9484bb9c75 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:55.520546 20344 tablet_bootstrap.cc:492] T ba5b4c81211647dd8383ec9484bb9c75 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap complete.
I20240502 21:28:55.520865 20344 ts_tablet_manager.cc:1367] T ba5b4c81211647dd8383ec9484bb9c75 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240502 21:28:55.521646 20344 raft_consensus.cc:348] T ba5b4c81211647dd8383ec9484bb9c75 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:28:55.521839 20344 raft_consensus.cc:729] T ba5b4c81211647dd8383ec9484bb9c75 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:28:55.522058 20344 consensus_queue.cc:260] T ba5b4c81211647dd8383ec9484bb9c75 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:28:55.522753 20344 ts_tablet_manager.cc:1398] T ba5b4c81211647dd8383ec9484bb9c75 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:28:55.523669 7505 tablet_copy_service.cc:342] P 31a72286397c4edc9a63eca3412be168: Request end of tablet copy session 73a1f2ca1f544de9b0e3b37015f85328-ba5b4c81211647dd8383ec9484bb9c75 received from {username='slave'} at 127.29.82.68:47963
I20240502 21:28:55.523798 7505 tablet_copy_service.cc:433] P 31a72286397c4edc9a63eca3412be168: ending tablet copy session 73a1f2ca1f544de9b0e3b37015f85328-ba5b4c81211647dd8383ec9484bb9c75 on tablet ba5b4c81211647dd8383ec9484bb9c75 with peer 73a1f2ca1f544de9b0e3b37015f85328
I20240502 21:28:55.534480 4705 consensus_queue.cc:237] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:28:55.537595 4574 raft_consensus.cc:1261] T a607fad427764c46b56f8963952d7613 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:28:55.537683 4440 raft_consensus.cc:1261] T a607fad427764c46b56f8963952d7613 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:28:55.538249 19918 consensus_queue.cc:1035] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:55.538581 19917 consensus_queue.cc:1035] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:28:55.538930 4641 consensus_peers.cc:485] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb -> Peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Couldn't send request to peer d0456c5e55a542128688904e003d403c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a607fad427764c46b56f8963952d7613. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:55.541431 19918 raft_consensus.cc:2932] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:55.542008 4574 raft_consensus.cc:2932] T a607fad427764c46b56f8963952d7613 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:55.546232 4440 raft_consensus.cc:2932] T a607fad427764c46b56f8963952d7613 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:55.551265 4293 catalog_manager.cc:5484] T a607fad427764c46b56f8963952d7613 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: config changed from index -1 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:55.579063 20389 ts_tablet_manager.cc:902] T 6687a533112445dc86951fc778d0725b P d0456c5e55a542128688904e003d403c: Initiating tablet copy from peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:28:55.580500 20389 tablet_copy_client.cc:308] T 6687a533112445dc86951fc778d0725b P d0456c5e55a542128688904e003d403c: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.65:42781
I20240502 21:28:55.581800 4463 tablet_copy_service.cc:140] P 71e1375b799b439fa3c83ca763c039c1: Received BeginTabletCopySession request for tablet 6687a533112445dc86951fc778d0725b from peer d0456c5e55a542128688904e003d403c ({username='slave'} at 127.29.82.70:41689)
I20240502 21:28:55.581966 4463 tablet_copy_service.cc:161] P 71e1375b799b439fa3c83ca763c039c1: Beginning new tablet copy session on tablet 6687a533112445dc86951fc778d0725b from peer d0456c5e55a542128688904e003d403c at {username='slave'} at 127.29.82.70:41689: session id = d0456c5e55a542128688904e003d403c-6687a533112445dc86951fc778d0725b
I20240502 21:28:55.585379 4463 tablet_copy_source_session.cc:205] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:28:55.587841 20389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6687a533112445dc86951fc778d0725b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:55.594902 20389 tablet_copy_client.cc:783] T 6687a533112445dc86951fc778d0725b P d0456c5e55a542128688904e003d403c: tablet copy: Starting download of 5 data blocks...
I20240502 21:28:55.605860 20389 tablet_copy_client.cc:647] T 6687a533112445dc86951fc778d0725b P d0456c5e55a542128688904e003d403c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:55.608755 20389 tablet_copy_client.cc:523] T 6687a533112445dc86951fc778d0725b P d0456c5e55a542128688904e003d403c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:55.611774 20407 ts_tablet_manager.cc:902] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 71e1375b799b439fa3c83ca763c039c1: Initiating tablet copy from peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:28:55.611985 20389 tablet_bootstrap.cc:492] T 6687a533112445dc86951fc778d0725b P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:28:55.624572 20407 tablet_copy_client.cc:308] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.70:40719
I20240502 21:28:55.625406 7370 tablet_copy_service.cc:140] P d0456c5e55a542128688904e003d403c: Received BeginTabletCopySession request for tablet d095a5e8f1f34b5ea2b6797ef06f79c6 from peer 71e1375b799b439fa3c83ca763c039c1 ({username='slave'} at 127.29.82.65:53175)
I20240502 21:28:55.625576 7370 tablet_copy_service.cc:161] P d0456c5e55a542128688904e003d403c: Beginning new tablet copy session on tablet d095a5e8f1f34b5ea2b6797ef06f79c6 from peer 71e1375b799b439fa3c83ca763c039c1 at {username='slave'} at 127.29.82.65:53175: session id = 71e1375b799b439fa3c83ca763c039c1-d095a5e8f1f34b5ea2b6797ef06f79c6
I20240502 21:28:55.628382 7370 tablet_copy_source_session.cc:205] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:55.631525 20407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d095a5e8f1f34b5ea2b6797ef06f79c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:55.637511 20389 tablet_bootstrap.cc:492] T 6687a533112445dc86951fc778d0725b P d0456c5e55a542128688904e003d403c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:55.637984 20389 tablet_bootstrap.cc:492] T 6687a533112445dc86951fc778d0725b P d0456c5e55a542128688904e003d403c: Bootstrap complete.
I20240502 21:28:55.638156 20407 tablet_copy_client.cc:783] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:55.638281 20389 ts_tablet_manager.cc:1367] T 6687a533112445dc86951fc778d0725b P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.026s user 0.018s sys 0.005s
I20240502 21:28:55.638350 20407 tablet_copy_client.cc:647] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:55.639146 20389 raft_consensus.cc:348] T 6687a533112445dc86951fc778d0725b P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:28:55.639357 20389 raft_consensus.cc:729] T 6687a533112445dc86951fc778d0725b P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:28:55.639678 20389 consensus_queue.cc:260] T 6687a533112445dc86951fc778d0725b P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:28:55.640332 20389 ts_tablet_manager.cc:1398] T 6687a533112445dc86951fc778d0725b P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:28:55.644064 4463 tablet_copy_service.cc:342] P 71e1375b799b439fa3c83ca763c039c1: Request end of tablet copy session d0456c5e55a542128688904e003d403c-6687a533112445dc86951fc778d0725b received from {username='slave'} at 127.29.82.70:41689
I20240502 21:28:55.644166 4463 tablet_copy_service.cc:433] P 71e1375b799b439fa3c83ca763c039c1: ending tablet copy session d0456c5e55a542128688904e003d403c-6687a533112445dc86951fc778d0725b on tablet 6687a533112445dc86951fc778d0725b with peer d0456c5e55a542128688904e003d403c
I20240502 21:28:55.653339 20407 tablet_copy_client.cc:523] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:55.656423 20407 tablet_bootstrap.cc:492] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:28:55.677419 20407 tablet_bootstrap.cc:492] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:55.677901 20407 tablet_bootstrap.cc:492] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap complete.
I20240502 21:28:55.678246 20407 ts_tablet_manager.cc:1367] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.002s
I20240502 21:28:55.678968 20407 raft_consensus.cc:348] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:28:55.679188 20407 raft_consensus.cc:729] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:28:55.679397 20407 consensus_queue.cc:260] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:28:55.680141 20407 ts_tablet_manager.cc:1398] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:28:55.681685 7370 tablet_copy_service.cc:342] P d0456c5e55a542128688904e003d403c: Request end of tablet copy session 71e1375b799b439fa3c83ca763c039c1-d095a5e8f1f34b5ea2b6797ef06f79c6 received from {username='slave'} at 127.29.82.65:53175
I20240502 21:28:55.681811 7370 tablet_copy_service.cc:433] P d0456c5e55a542128688904e003d403c: ending tablet copy session 71e1375b799b439fa3c83ca763c039c1-d095a5e8f1f34b5ea2b6797ef06f79c6 on tablet d095a5e8f1f34b5ea2b6797ef06f79c6 with peer 71e1375b799b439fa3c83ca763c039c1
I20240502 21:28:55.685292 20420 ts_tablet_manager.cc:902] T b5099940a48f4466aa7c713bb88d108e P 39981a8dfe94462b9a73ba489af4c714: Initiating tablet copy from peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:28:55.685464 20422 ts_tablet_manager.cc:902] T 686a74b8c1034112b293c4c7c6bcd99a P 398c3c8f3195492c900439ad5e1931bb: Initiating tablet copy from peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:28:55.699002 4836 raft_consensus.cc:1203] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:55.709062 20407 ts_tablet_manager.cc:902] T 0c56940914cb4161bd258bf881f06c07 P 71e1375b799b439fa3c83ca763c039c1: Initiating tablet copy from peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:28:55.742818 20420 tablet_copy_client.cc:308] T b5099940a48f4466aa7c713bb88d108e P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.68:42449
I20240502 21:28:55.743486 4859 tablet_copy_service.cc:140] P 73a1f2ca1f544de9b0e3b37015f85328: Received BeginTabletCopySession request for tablet b5099940a48f4466aa7c713bb88d108e from peer 39981a8dfe94462b9a73ba489af4c714 ({username='slave'} at 127.29.82.69:57807)
I20240502 21:28:55.743690 4859 tablet_copy_service.cc:161] P 73a1f2ca1f544de9b0e3b37015f85328: Beginning new tablet copy session on tablet b5099940a48f4466aa7c713bb88d108e from peer 39981a8dfe94462b9a73ba489af4c714 at {username='slave'} at 127.29.82.69:57807: session id = 39981a8dfe94462b9a73ba489af4c714-b5099940a48f4466aa7c713bb88d108e
I20240502 21:28:55.746546 4859 tablet_copy_source_session.cc:205] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:28:55.749213 20407 tablet_copy_client.cc:308] T 0c56940914cb4161bd258bf881f06c07 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.68:42449
I20240502 21:28:55.749874 4859 tablet_copy_service.cc:140] P 73a1f2ca1f544de9b0e3b37015f85328: Received BeginTabletCopySession request for tablet 0c56940914cb4161bd258bf881f06c07 from peer 71e1375b799b439fa3c83ca763c039c1 ({username='slave'} at 127.29.82.65:44725)
I20240502 21:28:55.750023 4859 tablet_copy_service.cc:161] P 73a1f2ca1f544de9b0e3b37015f85328: Beginning new tablet copy session on tablet 0c56940914cb4161bd258bf881f06c07 from peer 71e1375b799b439fa3c83ca763c039c1 at {username='slave'} at 127.29.82.65:44725: session id = 71e1375b799b439fa3c83ca763c039c1-0c56940914cb4161bd258bf881f06c07
I20240502 21:28:55.750895 20420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5099940a48f4466aa7c713bb88d108e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:55.752974 4859 tablet_copy_source_session.cc:205] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:55.754784 20407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c56940914cb4161bd258bf881f06c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:55.757015 20422 tablet_copy_client.cc:308] T 686a74b8c1034112b293c4c7c6bcd99a P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.68:42449
I20240502 21:28:55.757700 4859 tablet_copy_service.cc:140] P 73a1f2ca1f544de9b0e3b37015f85328: Received BeginTabletCopySession request for tablet 686a74b8c1034112b293c4c7c6bcd99a from peer 398c3c8f3195492c900439ad5e1931bb ({username='slave'} at 127.29.82.67:42171)
I20240502 21:28:55.757848 4859 tablet_copy_service.cc:161] P 73a1f2ca1f544de9b0e3b37015f85328: Beginning new tablet copy session on tablet 686a74b8c1034112b293c4c7c6bcd99a from peer 398c3c8f3195492c900439ad5e1931bb at {username='slave'} at 127.29.82.67:42171: session id = 398c3c8f3195492c900439ad5e1931bb-686a74b8c1034112b293c4c7c6bcd99a
I20240502 21:28:55.760694 4859 tablet_copy_source_session.cc:205] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:55.762184 20422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 686a74b8c1034112b293c4c7c6bcd99a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:55.763063 20420 tablet_copy_client.cc:783] T b5099940a48f4466aa7c713bb88d108e P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Starting download of 5 data blocks...
I20240502 21:28:55.764586 20407 tablet_copy_client.cc:783] T 0c56940914cb4161bd258bf881f06c07 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:55.764819 20407 tablet_copy_client.cc:647] T 0c56940914cb4161bd258bf881f06c07 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:55.766302 20407 tablet_copy_client.cc:523] T 0c56940914cb4161bd258bf881f06c07 P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:55.769197 20422 tablet_copy_client.cc:783] T 686a74b8c1034112b293c4c7c6bcd99a P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:55.769459 20422 tablet_copy_client.cc:647] T 686a74b8c1034112b293c4c7c6bcd99a P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:55.770928 20422 tablet_copy_client.cc:523] T 686a74b8c1034112b293c4c7c6bcd99a P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:55.771179 20407 tablet_bootstrap.cc:492] T 0c56940914cb4161bd258bf881f06c07 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:28:55.775624 20422 tablet_bootstrap.cc:492] T 686a74b8c1034112b293c4c7c6bcd99a P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:28:55.778916 20420 tablet_copy_client.cc:647] T b5099940a48f4466aa7c713bb88d108e P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:55.780771 20420 tablet_copy_client.cc:523] T b5099940a48f4466aa7c713bb88d108e P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:55.784518 20420 tablet_bootstrap.cc:492] T b5099940a48f4466aa7c713bb88d108e P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:28:55.797911 20407 tablet_bootstrap.cc:492] T 0c56940914cb4161bd258bf881f06c07 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:55.798422 20407 tablet_bootstrap.cc:492] T 0c56940914cb4161bd258bf881f06c07 P 71e1375b799b439fa3c83ca763c039c1: Bootstrap complete.
I20240502 21:28:55.798772 20407 ts_tablet_manager.cc:1367] T 0c56940914cb4161bd258bf881f06c07 P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.002s
I20240502 21:28:55.800896 20407 raft_consensus.cc:348] T 0c56940914cb4161bd258bf881f06c07 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:28:55.801162 20407 raft_consensus.cc:729] T 0c56940914cb4161bd258bf881f06c07 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:28:55.801424 20407 consensus_queue.cc:260] T 0c56940914cb4161bd258bf881f06c07 P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:28:55.802163 20407 ts_tablet_manager.cc:1398] T 0c56940914cb4161bd258bf881f06c07 P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:28:55.806351 4859 tablet_copy_service.cc:342] P 73a1f2ca1f544de9b0e3b37015f85328: Request end of tablet copy session 71e1375b799b439fa3c83ca763c039c1-0c56940914cb4161bd258bf881f06c07 received from {username='slave'} at 127.29.82.65:44725
I20240502 21:28:55.806563 4859 tablet_copy_service.cc:433] P 73a1f2ca1f544de9b0e3b37015f85328: ending tablet copy session 71e1375b799b439fa3c83ca763c039c1-0c56940914cb4161bd258bf881f06c07 on tablet 0c56940914cb4161bd258bf881f06c07 with peer 71e1375b799b439fa3c83ca763c039c1
I20240502 21:28:55.816742 20422 tablet_bootstrap.cc:492] T 686a74b8c1034112b293c4c7c6bcd99a P 398c3c8f3195492c900439ad5e1931bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:55.817173 20422 tablet_bootstrap.cc:492] T 686a74b8c1034112b293c4c7c6bcd99a P 398c3c8f3195492c900439ad5e1931bb: Bootstrap complete.
I20240502 21:28:55.817461 20422 ts_tablet_manager.cc:1367] T 686a74b8c1034112b293c4c7c6bcd99a P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.042s user 0.009s sys 0.007s
I20240502 21:28:55.818271 20422 raft_consensus.cc:348] T 686a74b8c1034112b293c4c7c6bcd99a P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:55.818562 20422 raft_consensus.cc:729] T 686a74b8c1034112b293c4c7c6bcd99a P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:28:55.822449 20422 consensus_queue.cc:260] T 686a74b8c1034112b293c4c7c6bcd99a P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:55.824036 20422 ts_tablet_manager.cc:1398] T 686a74b8c1034112b293c4c7c6bcd99a P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20240502 21:28:55.824683 4859 tablet_copy_service.cc:342] P 73a1f2ca1f544de9b0e3b37015f85328: Request end of tablet copy session 398c3c8f3195492c900439ad5e1931bb-686a74b8c1034112b293c4c7c6bcd99a received from {username='slave'} at 127.29.82.67:42171
I20240502 21:28:55.824808 4859 tablet_copy_service.cc:433] P 73a1f2ca1f544de9b0e3b37015f85328: ending tablet copy session 398c3c8f3195492c900439ad5e1931bb-686a74b8c1034112b293c4c7c6bcd99a on tablet 686a74b8c1034112b293c4c7c6bcd99a with peer 398c3c8f3195492c900439ad5e1931bb
I20240502 21:28:55.841532 20420 tablet_bootstrap.cc:492] T b5099940a48f4466aa7c713bb88d108e P 39981a8dfe94462b9a73ba489af4c714: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:55.847162 20420 tablet_bootstrap.cc:492] T b5099940a48f4466aa7c713bb88d108e P 39981a8dfe94462b9a73ba489af4c714: Bootstrap complete.
I20240502 21:28:55.847540 20420 ts_tablet_manager.cc:1367] T b5099940a48f4466aa7c713bb88d108e P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.063s user 0.018s sys 0.019s
I20240502 21:28:55.848625 20420 raft_consensus.cc:348] T b5099940a48f4466aa7c713bb88d108e P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:28:55.848963 20420 raft_consensus.cc:729] T b5099940a48f4466aa7c713bb88d108e P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:28:55.849323 20420 consensus_queue.cc:260] T b5099940a48f4466aa7c713bb88d108e P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:28:55.850108 20420 ts_tablet_manager.cc:1398] T b5099940a48f4466aa7c713bb88d108e P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:28:55.851298 4859 tablet_copy_service.cc:342] P 73a1f2ca1f544de9b0e3b37015f85328: Request end of tablet copy session 39981a8dfe94462b9a73ba489af4c714-b5099940a48f4466aa7c713bb88d108e received from {username='slave'} at 127.29.82.69:57807
I20240502 21:28:55.851444 4859 tablet_copy_service.cc:433] P 73a1f2ca1f544de9b0e3b37015f85328: ending tablet copy session 39981a8dfe94462b9a73ba489af4c714-b5099940a48f4466aa7c713bb88d108e on tablet b5099940a48f4466aa7c713bb88d108e with peer 39981a8dfe94462b9a73ba489af4c714
I20240502 21:28:55.867094 7396 maintenance_manager.cc:419] P d0456c5e55a542128688904e003d403c: Scheduling UndoDeltaBlockGCOp(6687a533112445dc86951fc778d0725b): 4841 bytes on disk
I20240502 21:28:55.868870 7286 maintenance_manager.cc:643] P d0456c5e55a542128688904e003d403c: UndoDeltaBlockGCOp(6687a533112445dc86951fc778d0725b) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
I20240502 21:28:55.903470 7262 maintenance_manager.cc:419] P 39981a8dfe94462b9a73ba489af4c714: Scheduling UndoDeltaBlockGCOp(b5099940a48f4466aa7c713bb88d108e): 4586 bytes on disk
I20240502 21:28:55.904218 7152 maintenance_manager.cc:643] P 39981a8dfe94462b9a73ba489af4c714: UndoDeltaBlockGCOp(b5099940a48f4466aa7c713bb88d108e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
W20240502 21:28:55.927394 7150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.70:40719, user_credentials={real_user=slave}} blocked reactor thread for 63652.6us
I20240502 21:28:55.965579 20389 ts_tablet_manager.cc:902] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c: Initiating tablet copy from peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:28:55.965705 20451 ts_tablet_manager.cc:902] T 6065bdba6e2840b8b368d8f92d1fa260 P 7a736a8ebb164ce0a051c0fccd76c3b8: Initiating tablet copy from peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:28:55.981671 20451 tablet_copy_client.cc:308] T 6065bdba6e2840b8b368d8f92d1fa260 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.72:41223
I20240502 21:28:55.982374 7638 tablet_copy_service.cc:140] P 0ddd486109f5434cace13d60d633ac04: Received BeginTabletCopySession request for tablet 6065bdba6e2840b8b368d8f92d1fa260 from peer 7a736a8ebb164ce0a051c0fccd76c3b8 ({username='slave'} at 127.29.82.66:56819)
I20240502 21:28:55.982563 7638 tablet_copy_service.cc:161] P 0ddd486109f5434cace13d60d633ac04: Beginning new tablet copy session on tablet 6065bdba6e2840b8b368d8f92d1fa260 from peer 7a736a8ebb164ce0a051c0fccd76c3b8 at {username='slave'} at 127.29.82.66:56819: session id = 7a736a8ebb164ce0a051c0fccd76c3b8-6065bdba6e2840b8b368d8f92d1fa260
I20240502 21:28:55.985520 7638 tablet_copy_source_session.cc:205] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:55.996062 20451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6065bdba6e2840b8b368d8f92d1fa260. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.003947 20451 tablet_copy_client.cc:783] T 6065bdba6e2840b8b368d8f92d1fa260 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:56.004276 20451 tablet_copy_client.cc:647] T 6065bdba6e2840b8b368d8f92d1fa260 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:56.005934 20451 tablet_copy_client.cc:523] T 6065bdba6e2840b8b368d8f92d1fa260 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:56.008788 20451 tablet_bootstrap.cc:492] T 6065bdba6e2840b8b368d8f92d1fa260 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:28:56.011971 20389 tablet_copy_client.cc:308] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.67:34537
I20240502 21:28:56.012781 4727 tablet_copy_service.cc:140] P 398c3c8f3195492c900439ad5e1931bb: Received BeginTabletCopySession request for tablet a607fad427764c46b56f8963952d7613 from peer d0456c5e55a542128688904e003d403c ({username='slave'} at 127.29.82.70:43047)
I20240502 21:28:56.012959 4727 tablet_copy_service.cc:161] P 398c3c8f3195492c900439ad5e1931bb: Beginning new tablet copy session on tablet a607fad427764c46b56f8963952d7613 from peer d0456c5e55a542128688904e003d403c at {username='slave'} at 127.29.82.70:43047: session id = d0456c5e55a542128688904e003d403c-a607fad427764c46b56f8963952d7613
I20240502 21:28:56.019261 20451 tablet_bootstrap.cc:492] T 6065bdba6e2840b8b368d8f92d1fa260 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:56.019749 20451 tablet_bootstrap.cc:492] T 6065bdba6e2840b8b368d8f92d1fa260 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap complete.
I20240502 21:28:56.020051 20451 ts_tablet_manager.cc:1367] T 6065bdba6e2840b8b368d8f92d1fa260 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.002s
I20240502 21:28:56.021265 20451 raft_consensus.cc:348] T 6065bdba6e2840b8b368d8f92d1fa260 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:28:56.021518 20451 raft_consensus.cc:729] T 6065bdba6e2840b8b368d8f92d1fa260 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.021826 20451 consensus_queue.cc:260] T 6065bdba6e2840b8b368d8f92d1fa260 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:28:56.022446 20451 ts_tablet_manager.cc:1398] T 6065bdba6e2840b8b368d8f92d1fa260 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:28:56.023288 7638 tablet_copy_service.cc:342] P 0ddd486109f5434cace13d60d633ac04: Request end of tablet copy session 7a736a8ebb164ce0a051c0fccd76c3b8-6065bdba6e2840b8b368d8f92d1fa260 received from {username='slave'} at 127.29.82.66:56819
I20240502 21:28:56.023418 7638 tablet_copy_service.cc:433] P 0ddd486109f5434cace13d60d633ac04: ending tablet copy session 7a736a8ebb164ce0a051c0fccd76c3b8-6065bdba6e2840b8b368d8f92d1fa260 on tablet 6065bdba6e2840b8b368d8f92d1fa260 with peer 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:28:56.035486 4727 tablet_copy_source_session.cc:205] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:28:56.037204 20389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a607fad427764c46b56f8963952d7613. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.046288 20389 tablet_copy_client.cc:783] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c: tablet copy: Starting download of 5 data blocks...
I20240502 21:28:56.072089 20389 tablet_copy_client.cc:647] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:56.072765 4836 raft_consensus.cc:1203] T ba5b4c81211647dd8383ec9484bb9c75 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:56.075353 20389 tablet_copy_client.cc:523] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:56.088811 20389 tablet_bootstrap.cc:492] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:28:56.137143 7347 raft_consensus.cc:1203] T 6687a533112445dc86951fc778d0725b P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:28:56.151129 4440 raft_consensus.cc:1203] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:56.174191 4440 raft_consensus.cc:1203] T 0c56940914cb4161bd258bf881f06c07 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:56.193161 20389 tablet_bootstrap.cc:492] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:56.198542 20389 tablet_bootstrap.cc:492] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c: Bootstrap complete.
I20240502 21:28:56.198989 20389 ts_tablet_manager.cc:1367] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.110s user 0.017s sys 0.015s
I20240502 21:28:56.202343 20389 raft_consensus.cc:348] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:28:56.202708 20389 raft_consensus.cc:729] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.203011 20389 consensus_queue.cc:260] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:28:56.203846 20389 ts_tablet_manager.cc:1398] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:28:56.204635 4727 tablet_copy_service.cc:342] P 398c3c8f3195492c900439ad5e1931bb: Request end of tablet copy session d0456c5e55a542128688904e003d403c-a607fad427764c46b56f8963952d7613 received from {username='slave'} at 127.29.82.70:43047
I20240502 21:28:56.204795 4727 tablet_copy_service.cc:433] P 398c3c8f3195492c900439ad5e1931bb: ending tablet copy session d0456c5e55a542128688904e003d403c-a607fad427764c46b56f8963952d7613 on tablet a607fad427764c46b56f8963952d7613 with peer d0456c5e55a542128688904e003d403c
W20240502 21:28:56.235481 4773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.66:35781, user_credentials={real_user=slave}} blocked reactor thread for 65107.1us
I20240502 21:28:56.240129 4706 raft_consensus.cc:1203] T 686a74b8c1034112b293c4c7c6bcd99a P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:56.253814 7214 raft_consensus.cc:1203] T b5099940a48f4466aa7c713bb88d108e P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:28:56.277426 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:56.314301 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:56.349016 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:56.377406 7396 maintenance_manager.cc:419] P d0456c5e55a542128688904e003d403c: Scheduling UndoDeltaBlockGCOp(a607fad427764c46b56f8963952d7613): 4839 bytes on disk
I20240502 21:28:56.378342 7286 maintenance_manager.cc:643] P d0456c5e55a542128688904e003d403c: UndoDeltaBlockGCOp(a607fad427764c46b56f8963952d7613) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240502 21:28:56.396872 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:56.409950 7347 raft_consensus.cc:1203] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:28:56.410562 4575 raft_consensus.cc:1203] T 6065bdba6e2840b8b368d8f92d1fa260 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:28:56.419559 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:56.485493 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:56.539863 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:56.562453 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:56.713176 4836 consensus_queue.cc:237] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:28:56.715152 7214 raft_consensus.cc:1261] T b5099940a48f4466aa7c713bb88d108e P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:56.715200 4440 raft_consensus.cc:1261] T b5099940a48f4466aa7c713bb88d108e P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:56.715759 20503 consensus_queue.cc:1035] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:56.716002 20504 consensus_queue.cc:1035] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:56.718508 20503 raft_consensus.cc:2932] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:56.719184 7214 raft_consensus.cc:2932] T b5099940a48f4466aa7c713bb88d108e P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:56.719962 4440 raft_consensus.cc:2932] T b5099940a48f4466aa7c713bb88d108e P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:56.726233 4293 catalog_manager.cc:5484] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:28:56.753491 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet b5099940a48f4466aa7c713bb88d108e with delete_type TABLET_DATA_TOMBSTONED (TS 398c3c8f3195492c900439ad5e1931bb not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44764
I20240502 21:28:56.756543 20540 tablet_replica.cc:330] T b5099940a48f4466aa7c713bb88d108e P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:28:56.756755 20540 raft_consensus.cc:2229] T b5099940a48f4466aa7c713bb88d108e P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:56.757005 20540 raft_consensus.cc:2258] T b5099940a48f4466aa7c713bb88d108e P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:56.757822 20540 ts_tablet_manager.cc:1875] T b5099940a48f4466aa7c713bb88d108e P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:56.786604 20540 ts_tablet_manager.cc:1888] T b5099940a48f4466aa7c713bb88d108e P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:28:56.786749 20540 log.cc:1196] T b5099940a48f4466aa7c713bb88d108e P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/b5099940a48f4466aa7c713bb88d108e
I20240502 21:28:56.787536 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet b5099940a48f4466aa7c713bb88d108e (table rebalance_test_table_3 [id=3c51dfaf2d824f93813d8d39758c9a8c]) successfully deleted
I20240502 21:28:57.134184 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:57.148352 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:57.172327 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:57.220021 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:57.229846 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:57.264391 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:57.274704 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:57.275414 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:57.498682 4836 consensus_queue.cc:237] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:28:57.500552 4706 raft_consensus.cc:1261] T 686a74b8c1034112b293c4c7c6bcd99a P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:57.500762 7483 raft_consensus.cc:1261] T 686a74b8c1034112b293c4c7c6bcd99a P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:57.501101 20504 consensus_queue.cc:1035] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:57.501399 20485 consensus_queue.cc:1035] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:57.504256 20504 raft_consensus.cc:2932] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:57.504618 4706 raft_consensus.cc:2932] T 686a74b8c1034112b293c4c7c6bcd99a P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:57.504771 7483 raft_consensus.cc:2932] T 686a74b8c1034112b293c4c7c6bcd99a P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:57.511195 4291 catalog_manager.cc:5484] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:28:57.517410 7328 tablet_service.cc:1514] Processing DeleteTablet for tablet 686a74b8c1034112b293c4c7c6bcd99a with delete_type TABLET_DATA_TOMBSTONED (TS d0456c5e55a542128688904e003d403c not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47078
I20240502 21:28:57.518119 20629 tablet_replica.cc:330] T 686a74b8c1034112b293c4c7c6bcd99a P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:28:57.518354 20629 raft_consensus.cc:2229] T 686a74b8c1034112b293c4c7c6bcd99a P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:57.518617 20629 raft_consensus.cc:2258] T 686a74b8c1034112b293c4c7c6bcd99a P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:57.519609 20629 ts_tablet_manager.cc:1875] T 686a74b8c1034112b293c4c7c6bcd99a P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:57.525698 20629 ts_tablet_manager.cc:1888] T 686a74b8c1034112b293c4c7c6bcd99a P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:57.525889 20629 log.cc:1196] T 686a74b8c1034112b293c4c7c6bcd99a P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/686a74b8c1034112b293c4c7c6bcd99a
I20240502 21:28:57.526928 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet 686a74b8c1034112b293c4c7c6bcd99a (table rebalancer_extra_table_16 [id=23c1b78fb75948c193bd4eb1c32f8bef]) successfully deleted
I20240502 21:28:57.906365 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:57.913468 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:57.920661 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:57.932152 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:57.973466 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:57.974707 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240502 21:28:57.981899 7151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.66:35781, user_credentials={real_user=slave}} blocked reactor thread for 59940.7us
I20240502 21:28:58.006580 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:58.069181 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:58.234405 7483 consensus_queue.cc:237] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:28:58.236196 4836 raft_consensus.cc:1261] T ba5b4c81211647dd8383ec9484bb9c75 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:58.236289 4705 raft_consensus.cc:1261] T ba5b4c81211647dd8383ec9484bb9c75 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:58.236800 20144 consensus_queue.cc:1035] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:58.237082 19854 consensus_queue.cc:1035] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:58.242738 20144 raft_consensus.cc:2932] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:58.259305 4705 raft_consensus.cc:2932] T ba5b4c81211647dd8383ec9484bb9c75 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:58.260272 4293 catalog_manager.cc:5484] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 reported cstate change: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:58.262447 4836 raft_consensus.cc:2932] T ba5b4c81211647dd8383ec9484bb9c75 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:28:58.270164 7597 tablet_service.cc:1514] Processing DeleteTablet for tablet ba5b4c81211647dd8383ec9484bb9c75 with delete_type TABLET_DATA_TOMBSTONED (TS 0ddd486109f5434cace13d60d633ac04 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44444
I20240502 21:28:58.286741 20721 tablet_replica.cc:330] T ba5b4c81211647dd8383ec9484bb9c75 P 0ddd486109f5434cace13d60d633ac04: stopping tablet replica
I20240502 21:28:58.287317 20721 raft_consensus.cc:2229] T ba5b4c81211647dd8383ec9484bb9c75 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:58.288386 20721 raft_consensus.cc:2258] T ba5b4c81211647dd8383ec9484bb9c75 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:58.289594 20721 ts_tablet_manager.cc:1875] T ba5b4c81211647dd8383ec9484bb9c75 P 0ddd486109f5434cace13d60d633ac04: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:58.303689 20721 ts_tablet_manager.cc:1888] T ba5b4c81211647dd8383ec9484bb9c75 P 0ddd486109f5434cace13d60d633ac04: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:58.303952 20721 log.cc:1196] T ba5b4c81211647dd8383ec9484bb9c75 P 0ddd486109f5434cace13d60d633ac04: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/wals/ba5b4c81211647dd8383ec9484bb9c75
I20240502 21:28:58.304739 4285 catalog_manager.cc:4830] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): tablet ba5b4c81211647dd8383ec9484bb9c75 (table rebalancer_extra_table_2 [id=1b458a4121624eaaaf47feadd84aab8c]) successfully deleted
I20240502 21:28:58.636495 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:58.679662 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:58.681558 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:58.694018 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:58.750766 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:58.751639 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:28:58.842963 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:58.842968 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:59.009742 7350 consensus_queue.cc:237] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:28:59.011799 7214 raft_consensus.cc:1261] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:59.012395 4440 raft_consensus.cc:1261] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:28:59.012352 20258 consensus_queue.cc:1035] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:59.012928 20258 consensus_queue.cc:1035] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:28:59.016355 20258 raft_consensus.cc:2932] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:59.017241 7214 raft_consensus.cc:2932] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:59.025897 4294 catalog_manager.cc:5484] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c reported cstate change: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:59.034731 4440 raft_consensus.cc:2932] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:28:59.058369 7597 tablet_service.cc:1514] Processing DeleteTablet for tablet d095a5e8f1f34b5ea2b6797ef06f79c6 with delete_type TABLET_DATA_TOMBSTONED (TS 0ddd486109f5434cace13d60d633ac04 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44444
I20240502 21:28:59.059046 20822 tablet_replica.cc:330] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 0ddd486109f5434cace13d60d633ac04: stopping tablet replica
I20240502 21:28:59.060882 20822 raft_consensus.cc:2229] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:59.061196 20822 raft_consensus.cc:2258] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:59.061906 20822 ts_tablet_manager.cc:1875] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 0ddd486109f5434cace13d60d633ac04: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:59.067520 20822 ts_tablet_manager.cc:1888] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 0ddd486109f5434cace13d60d633ac04: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:28:59.067708 20822 log.cc:1196] T d095a5e8f1f34b5ea2b6797ef06f79c6 P 0ddd486109f5434cace13d60d633ac04: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/wals/d095a5e8f1f34b5ea2b6797ef06f79c6
I20240502 21:28:59.068440 4285 catalog_manager.cc:4830] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): tablet d095a5e8f1f34b5ea2b6797ef06f79c6 (table rebalancer_extra_table_2 [id=1b458a4121624eaaaf47feadd84aab8c]) successfully deleted
I20240502 21:28:59.410660 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:28:59.419802 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:59.423431 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:59.464263 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:28:59.594318 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:28:59.617507 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 21:28:59.619393 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.71:33913, user_credentials={real_user=slave}} blocked reactor thread for 82053us
I20240502 21:28:59.669731 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:59.713415 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:28:59.879518 4706 consensus_queue.cc:237] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:28:59.881516 7349 raft_consensus.cc:1261] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:59.881516 4440 raft_consensus.cc:1261] T a607fad427764c46b56f8963952d7613 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:59.882078 20889 consensus_queue.cc:1035] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:59.882390 19917 consensus_queue.cc:1035] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:59.886699 19917 raft_consensus.cc:2932] T a607fad427764c46b56f8963952d7613 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:59.887297 7350 raft_consensus.cc:2932] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:59.895960 4299 catalog_manager.cc:5484] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c reported cstate change: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:59.908663 4440 raft_consensus.cc:2932] T a607fad427764c46b56f8963952d7613 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:28:59.928306 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet a607fad427764c46b56f8963952d7613 with delete_type TABLET_DATA_TOMBSTONED (TS 7a736a8ebb164ce0a051c0fccd76c3b8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60018
I20240502 21:28:59.928992 20918 tablet_replica.cc:330] T a607fad427764c46b56f8963952d7613 P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:28:59.929275 20918 raft_consensus.cc:2229] T a607fad427764c46b56f8963952d7613 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:59.929620 20918 raft_consensus.cc:2258] T a607fad427764c46b56f8963952d7613 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:59.930291 20918 ts_tablet_manager.cc:1875] T a607fad427764c46b56f8963952d7613 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:59.943854 20918 ts_tablet_manager.cc:1888] T a607fad427764c46b56f8963952d7613 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:28:59.944895 20918 log.cc:1196] T a607fad427764c46b56f8963952d7613 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/a607fad427764c46b56f8963952d7613
I20240502 21:28:59.945847 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet a607fad427764c46b56f8963952d7613 (table rebalance_test_table_9 [id=80b493b0fc3f49dcbf948cee9e748e3c]) successfully deleted
I20240502 21:29:00.302206 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:00.307091 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:00.338382 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:00.344678 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:00.349500 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:29:00.422910 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:29:00.443856 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:00.465927 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:00.647671 4440 consensus_queue.cc:237] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:29:00.650127 7348 raft_consensus.cc:1261] T 6687a533112445dc86951fc778d0725b P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:00.650749 20237 consensus_queue.cc:1035] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:00.651043 4705 raft_consensus.cc:1261] T 6687a533112445dc86951fc778d0725b P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:00.651811 20237 consensus_queue.cc:1035] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:00.654829 20236 raft_consensus.cc:2932] T 6687a533112445dc86951fc778d0725b P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:29:00.655398 4705 raft_consensus.cc:2932] T 6687a533112445dc86951fc778d0725b P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:29:00.661027 7348 raft_consensus.cc:2932] T 6687a533112445dc86951fc778d0725b P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:29:00.662982 4297 catalog_manager.cc:5484] T 6687a533112445dc86951fc778d0725b P 398c3c8f3195492c900439ad5e1931bb reported cstate change: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:29:00.698478 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet 6687a533112445dc86951fc778d0725b with delete_type TABLET_DATA_TOMBSTONED (TS 7a736a8ebb164ce0a051c0fccd76c3b8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60018
I20240502 21:29:00.699113 21020 tablet_replica.cc:330] T 6687a533112445dc86951fc778d0725b P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:29:00.699407 21020 raft_consensus.cc:2229] T 6687a533112445dc86951fc778d0725b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:00.700068 21020 raft_consensus.cc:2258] T 6687a533112445dc86951fc778d0725b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:00.700883 21020 ts_tablet_manager.cc:1875] T 6687a533112445dc86951fc778d0725b P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:00.718288 21020 ts_tablet_manager.cc:1888] T 6687a533112445dc86951fc778d0725b P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:29:00.718545 21020 log.cc:1196] T 6687a533112445dc86951fc778d0725b P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/6687a533112445dc86951fc778d0725b
I20240502 21:29:00.719734 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet 6687a533112445dc86951fc778d0725b (table rebalance_test_table_13 [id=d14fe34dcad44629af56956cc9e71e5b]) successfully deleted
I20240502 21:29:01.184446 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:01.188627 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:01.202147 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:29:01.248564 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:01.278251 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:01.312063 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 21:29:01.363890 4508 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.71:33913, user_credentials={real_user=slave}} blocked reactor thread for 46275us
I20240502 21:29:01.380056 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:01.408308 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:29:01.586000 4574 consensus_queue.cc:237] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:01.588589 4837 raft_consensus.cc:1261] T a33fd7d5a8db41a39ba780e16a84f9b1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:01.588651 7213 raft_consensus.cc:1261] T a33fd7d5a8db41a39ba780e16a84f9b1 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:01.589294 20793 consensus_queue.cc:1035] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:01.589665 20890 consensus_queue.cc:1035] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:01.592571 20793 raft_consensus.cc:2932] T a33fd7d5a8db41a39ba780e16a84f9b1 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:01.593571 7213 raft_consensus.cc:2932] T a33fd7d5a8db41a39ba780e16a84f9b1 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:01.596496 4837 raft_consensus.cc:2932] T a33fd7d5a8db41a39ba780e16a84f9b1 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:01.599539 4299 catalog_manager.cc:5484] T a33fd7d5a8db41a39ba780e16a84f9b1 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:01.609323 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet a33fd7d5a8db41a39ba780e16a84f9b1 with delete_type TABLET_DATA_TOMBSTONED (TS 398c3c8f3195492c900439ad5e1931bb not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44764
I20240502 21:29:01.609946 21129 tablet_replica.cc:330] T a33fd7d5a8db41a39ba780e16a84f9b1 P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:29:01.610282 21129 raft_consensus.cc:2229] T a33fd7d5a8db41a39ba780e16a84f9b1 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:01.610639 21129 raft_consensus.cc:2258] T a33fd7d5a8db41a39ba780e16a84f9b1 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:01.611706 21129 ts_tablet_manager.cc:1875] T a33fd7d5a8db41a39ba780e16a84f9b1 P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:01.627020 21129 ts_tablet_manager.cc:1888] T a33fd7d5a8db41a39ba780e16a84f9b1 P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:29:01.627142 21129 log.cc:1196] T a33fd7d5a8db41a39ba780e16a84f9b1 P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/a33fd7d5a8db41a39ba780e16a84f9b1
I20240502 21:29:01.628263 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet a33fd7d5a8db41a39ba780e16a84f9b1 (table rebalance_test_table_7 [id=ccfdd338c3a14e4eaab745d78ee6de47]) successfully deleted
I20240502 21:29:02.125547 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:02.126135 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:02.184507 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:02.193707 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:02.204229 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:29:02.230044 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:02.309223 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:02.325757 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:29:02.474200 7484 consensus_queue.cc:237] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:29:02.477682 4837 raft_consensus.cc:1261] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:29:02.477816 7213 raft_consensus.cc:1261] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:29:02.478261 20716 consensus_queue.cc:1035] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:02.478542 21099 consensus_queue.cc:1035] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:02.483769 20716 raft_consensus.cc:2932] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:29:02.484503 7213 raft_consensus.cc:2932] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:29:02.492043 4291 catalog_manager.cc:5484] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:29:02.506035 4836 raft_consensus.cc:2932] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:29:02.512411 7597 tablet_service.cc:1514] Processing DeleteTablet for tablet 063bdcf9cea74d5cbe3143eb16fba6a5 with delete_type TABLET_DATA_TOMBSTONED (TS 0ddd486109f5434cace13d60d633ac04 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44444
I20240502 21:29:02.513020 21230 tablet_replica.cc:330] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 0ddd486109f5434cace13d60d633ac04: stopping tablet replica
I20240502 21:29:02.513243 21230 raft_consensus.cc:2229] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:02.513427 21230 raft_consensus.cc:2258] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:02.514036 21230 ts_tablet_manager.cc:1875] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 0ddd486109f5434cace13d60d633ac04: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:02.520772 21230 ts_tablet_manager.cc:1888] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 0ddd486109f5434cace13d60d633ac04: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:29:02.520915 21230 log.cc:1196] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 0ddd486109f5434cace13d60d633ac04: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/wals/063bdcf9cea74d5cbe3143eb16fba6a5
I20240502 21:29:02.522027 4285 catalog_manager.cc:4830] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): tablet 063bdcf9cea74d5cbe3143eb16fba6a5 (table rebalancer_extra_table_18 [id=1488a4e68a204d09a88b4a05c1c2cc24]) successfully deleted
I20240502 21:29:02.884828 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:02.910027 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:29:02.926729 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:03.016129 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:03.022166 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:03.023527 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:03.033424 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:03.069226 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:29:03.224447 4836 consensus_queue.cc:237] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:29:03.226500 7213 raft_consensus.cc:1261] T 0c56940914cb4161bd258bf881f06c07 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:29:03.227032 20792 consensus_queue.cc:1035] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:03.227309 4440 raft_consensus.cc:1261] T 0c56940914cb4161bd258bf881f06c07 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:29:03.227835 20792 consensus_queue.cc:1035] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:03.230720 21101 raft_consensus.cc:2932] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:29:03.231400 7213 raft_consensus.cc:2932] T 0c56940914cb4161bd258bf881f06c07 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:29:03.231882 4440 raft_consensus.cc:2932] T 0c56940914cb4161bd258bf881f06c07 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:29:03.238294 4299 catalog_manager.cc:5484] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:29:03.280589 7597 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c56940914cb4161bd258bf881f06c07 with delete_type TABLET_DATA_TOMBSTONED (TS 0ddd486109f5434cace13d60d633ac04 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44444
I20240502 21:29:03.281159 21322 tablet_replica.cc:330] T 0c56940914cb4161bd258bf881f06c07 P 0ddd486109f5434cace13d60d633ac04: stopping tablet replica
I20240502 21:29:03.281381 21322 raft_consensus.cc:2229] T 0c56940914cb4161bd258bf881f06c07 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:03.281586 21322 raft_consensus.cc:2258] T 0c56940914cb4161bd258bf881f06c07 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:03.282218 21322 ts_tablet_manager.cc:1875] T 0c56940914cb4161bd258bf881f06c07 P 0ddd486109f5434cace13d60d633ac04: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:03.288396 21322 ts_tablet_manager.cc:1888] T 0c56940914cb4161bd258bf881f06c07 P 0ddd486109f5434cace13d60d633ac04: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:29:03.288563 21322 log.cc:1196] T 0c56940914cb4161bd258bf881f06c07 P 0ddd486109f5434cace13d60d633ac04: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/wals/0c56940914cb4161bd258bf881f06c07
I20240502 21:29:03.289261 4285 catalog_manager.cc:4830] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): tablet 0c56940914cb4161bd258bf881f06c07 (table rebalancer_extra_table_48 [id=2e1fa2b5e59045abbc77e0d878e997ac]) successfully deleted
I20240502 21:29:03.670720 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:03.692891 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:03.695024 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:29:03.736965 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:03.772663 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:03.775828 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:03.805137 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:29:03.847980 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:04.034291 7617 consensus_queue.cc:237] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:29:04.037168 4574 raft_consensus.cc:1261] T 6065bdba6e2840b8b368d8f92d1fa260 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:29:04.037678 19761 consensus_queue.cc:1035] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:04.040720 19761 raft_consensus.cc:2932] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:29:04.041271 4574 raft_consensus.cc:2932] T 6065bdba6e2840b8b368d8f92d1fa260 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:29:04.042193 7213 raft_consensus.cc:1261] T 6065bdba6e2840b8b368d8f92d1fa260 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:29:04.043673 21067 consensus_queue.cc:1035] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:04.046023 7213 raft_consensus.cc:2932] T 6065bdba6e2840b8b368d8f92d1fa260 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:29:04.047798 4299 catalog_manager.cc:5484] T 6065bdba6e2840b8b368d8f92d1fa260 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: config changed from index 2 to 3, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) evicted. New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:29:04.080458 7462 tablet_service.cc:1514] Processing DeleteTablet for tablet 6065bdba6e2840b8b368d8f92d1fa260 with delete_type TABLET_DATA_TOMBSTONED (TS 31a72286397c4edc9a63eca3412be168 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:37182
I20240502 21:29:04.097071 21434 tablet_replica.cc:330] T 6065bdba6e2840b8b368d8f92d1fa260 P 31a72286397c4edc9a63eca3412be168: stopping tablet replica
I20240502 21:29:04.097446 21434 raft_consensus.cc:2229] T 6065bdba6e2840b8b368d8f92d1fa260 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:04.097748 21434 raft_consensus.cc:2258] T 6065bdba6e2840b8b368d8f92d1fa260 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:04.098462 21434 ts_tablet_manager.cc:1875] T 6065bdba6e2840b8b368d8f92d1fa260 P 31a72286397c4edc9a63eca3412be168: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:04.105408 21434 ts_tablet_manager.cc:1888] T 6065bdba6e2840b8b368d8f92d1fa260 P 31a72286397c4edc9a63eca3412be168: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:29:04.105607 21434 log.cc:1196] T 6065bdba6e2840b8b368d8f92d1fa260 P 31a72286397c4edc9a63eca3412be168: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/wals/6065bdba6e2840b8b368d8f92d1fa260
I20240502 21:29:04.106617 4284 catalog_manager.cc:4830] TS 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): tablet 6065bdba6e2840b8b368d8f92d1fa260 (table rebalancer_extra_table_18 [id=1488a4e68a204d09a88b4a05c1c2cc24]) successfully deleted
I20240502 21:29:04.179884 7213 consensus_queue.cc:237] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
W20240502 21:29:04.182821 7148 consensus_peers.cc:485] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 -> Peer 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): Couldn't send request to peer 31a72286397c4edc9a63eca3412be168. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db24db8916314fdc8d9c8a3b09019604. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:04.182886 4574 raft_consensus.cc:1261] T db24db8916314fdc8d9c8a3b09019604 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:29:04.183189 7349 raft_consensus.cc:1261] T db24db8916314fdc8d9c8a3b09019604 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:29:04.183494 21456 consensus_queue.cc:1035] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:29:04.183986 20447 consensus_queue.cc:1035] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:29:04.187564 20447 raft_consensus.cc:2932] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:04.188191 7349 raft_consensus.cc:2932] T db24db8916314fdc8d9c8a3b09019604 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:04.191715 4574 raft_consensus.cc:2932] T db24db8916314fdc8d9c8a3b09019604 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:04.198508 4300 catalog_manager.cc:5484] T db24db8916314fdc8d9c8a3b09019604 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: config changed from index -1 to 2, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:04.233753 7349 consensus_queue.cc:237] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:29:04.236474 7617 raft_consensus.cc:1261] T 9a36933eaf04449faad747e5f44652b9 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:29:04.236938 7281 consensus_peers.cc:485] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a36933eaf04449faad747e5f44652b9. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:04.237061 7484 raft_consensus.cc:1261] T 9a36933eaf04449faad747e5f44652b9 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:29:04.237118 21293 consensus_queue.cc:1035] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:29:04.238821 21293 consensus_queue.cc:1035] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:29:04.241408 21184 raft_consensus.cc:2932] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:29:04.241921 7484 raft_consensus.cc:2932] T 9a36933eaf04449faad747e5f44652b9 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:29:04.252485 4297 catalog_manager.cc:5484] T 9a36933eaf04449faad747e5f44652b9 P 31a72286397c4edc9a63eca3412be168 reported cstate change: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:29:04.258514 7617 raft_consensus.cc:2932] T 9a36933eaf04449faad747e5f44652b9 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:29:04.296193 4573 consensus_queue.cc:237] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:04.299012 4704 raft_consensus.cc:1261] T f4cf59f48e874f42aed76d694246e7d3 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:29:04.299163 4510 consensus_peers.cc:485] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 -> Peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Couldn't send request to peer 39981a8dfe94462b9a73ba489af4c714. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4cf59f48e874f42aed76d694246e7d3. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:04.299283 4837 raft_consensus.cc:1261] T f4cf59f48e874f42aed76d694246e7d3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:29:04.299907 21496 consensus_queue.cc:1035] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:29:04.300220 21053 consensus_queue.cc:1035] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:29:04.303994 21053 raft_consensus.cc:2932] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:04.304594 4837 raft_consensus.cc:2932] T f4cf59f48e874f42aed76d694246e7d3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:04.310226 4706 raft_consensus.cc:2932] T f4cf59f48e874f42aed76d694246e7d3 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:04.313463 4299 catalog_manager.cc:5484] T f4cf59f48e874f42aed76d694246e7d3 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index -1 to 2, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:04.374805 4706 consensus_queue.cc:237] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:04.377737 7617 raft_consensus.cc:1261] T a4c36adfc86141fbb8724d5a6a798c26 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:29:04.378244 4573 raft_consensus.cc:1261] T a4c36adfc86141fbb8724d5a6a798c26 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:29:04.378612 4639 consensus_peers.cc:485] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb -> Peer 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): Couldn't send request to peer 31a72286397c4edc9a63eca3412be168. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4c36adfc86141fbb8724d5a6a798c26. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:04.380980 20889 consensus_queue.cc:1035] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:29:04.381497 20889 consensus_queue.cc:1035] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:29:04.408427 21516 raft_consensus.cc:2932] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:04.411160 4574 raft_consensus.cc:2932] T a4c36adfc86141fbb8724d5a6a798c26 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:04.421233 4299 catalog_manager.cc:5484] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: config changed from index -1 to 2, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:04.423251 7617 raft_consensus.cc:2932] T a4c36adfc86141fbb8724d5a6a798c26 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:04.478181 7617 consensus_queue.cc:237] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
W20240502 21:29:04.486264 7552 consensus_peers.cc:485] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25be332e341d4869b2165da105c17a8a. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:04.487251 4441 raft_consensus.cc:1261] T 25be332e341d4869b2165da105c17a8a P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:29:04.487931 19761 consensus_queue.cc:1035] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:29:04.490103 7349 raft_consensus.cc:1261] T 25be332e341d4869b2165da105c17a8a P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:29:04.491006 19761 consensus_queue.cc:1035] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:29:04.493814 21425 raft_consensus.cc:2932] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:29:04.494374 7349 raft_consensus.cc:2932] T 25be332e341d4869b2165da105c17a8a P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:29:04.502621 4299 catalog_manager.cc:5484] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 reported cstate change: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:29:04.502969 4441 raft_consensus.cc:2932] T 25be332e341d4869b2165da105c17a8a P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:29:04.697029 21581 ts_tablet_manager.cc:902] T db24db8916314fdc8d9c8a3b09019604 P 31a72286397c4edc9a63eca3412be168: Initiating tablet copy from peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:29:04.701440 21581 tablet_copy_client.cc:308] T db24db8916314fdc8d9c8a3b09019604 P 31a72286397c4edc9a63eca3412be168: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.69:33193
I20240502 21:29:04.702064 7236 tablet_copy_service.cc:140] P 39981a8dfe94462b9a73ba489af4c714: Received BeginTabletCopySession request for tablet db24db8916314fdc8d9c8a3b09019604 from peer 31a72286397c4edc9a63eca3412be168 ({username='slave'} at 127.29.82.71:46939)
I20240502 21:29:04.702247 7236 tablet_copy_service.cc:161] P 39981a8dfe94462b9a73ba489af4c714: Beginning new tablet copy session on tablet db24db8916314fdc8d9c8a3b09019604 from peer 31a72286397c4edc9a63eca3412be168 at {username='slave'} at 127.29.82.71:46939: session id = 31a72286397c4edc9a63eca3412be168-db24db8916314fdc8d9c8a3b09019604
I20240502 21:29:04.705222 7236 tablet_copy_source_session.cc:205] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:04.717466 21581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db24db8916314fdc8d9c8a3b09019604. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:04.741633 21581 tablet_copy_client.cc:783] T db24db8916314fdc8d9c8a3b09019604 P 31a72286397c4edc9a63eca3412be168: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:04.741945 21581 tablet_copy_client.cc:647] T db24db8916314fdc8d9c8a3b09019604 P 31a72286397c4edc9a63eca3412be168: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:04.760128 21581 tablet_copy_client.cc:523] T db24db8916314fdc8d9c8a3b09019604 P 31a72286397c4edc9a63eca3412be168: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:04.766287 21589 ts_tablet_manager.cc:902] T 9a36933eaf04449faad747e5f44652b9 P 73a1f2ca1f544de9b0e3b37015f85328: Initiating tablet copy from peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:29:04.771826 21581 tablet_bootstrap.cc:492] T db24db8916314fdc8d9c8a3b09019604 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:29:04.791096 21589 tablet_copy_client.cc:308] T 9a36933eaf04449faad747e5f44652b9 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.70:40719
I20240502 21:29:04.791838 7370 tablet_copy_service.cc:140] P d0456c5e55a542128688904e003d403c: Received BeginTabletCopySession request for tablet 9a36933eaf04449faad747e5f44652b9 from peer 73a1f2ca1f544de9b0e3b37015f85328 ({username='slave'} at 127.29.82.68:49699)
I20240502 21:29:04.792063 7370 tablet_copy_service.cc:161] P d0456c5e55a542128688904e003d403c: Beginning new tablet copy session on tablet 9a36933eaf04449faad747e5f44652b9 from peer 73a1f2ca1f544de9b0e3b37015f85328 at {username='slave'} at 127.29.82.68:49699: session id = 73a1f2ca1f544de9b0e3b37015f85328-9a36933eaf04449faad747e5f44652b9
I20240502 21:29:04.794899 7370 tablet_copy_source_session.cc:205] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:04.796226 21589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a36933eaf04449faad747e5f44652b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:04.803102 21589 tablet_copy_client.cc:783] T 9a36933eaf04449faad747e5f44652b9 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:04.803316 21589 tablet_copy_client.cc:647] T 9a36933eaf04449faad747e5f44652b9 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:04.805008 21589 tablet_copy_client.cc:523] T 9a36933eaf04449faad747e5f44652b9 P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:04.808871 21589 tablet_bootstrap.cc:492] T 9a36933eaf04449faad747e5f44652b9 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:29:04.810672 21581 tablet_bootstrap.cc:492] T db24db8916314fdc8d9c8a3b09019604 P 31a72286397c4edc9a63eca3412be168: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:04.811107 21581 tablet_bootstrap.cc:492] T db24db8916314fdc8d9c8a3b09019604 P 31a72286397c4edc9a63eca3412be168: Bootstrap complete.
I20240502 21:29:04.811364 21581 ts_tablet_manager.cc:1367] T db24db8916314fdc8d9c8a3b09019604 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.005s
I20240502 21:29:04.812176 21581 raft_consensus.cc:348] T db24db8916314fdc8d9c8a3b09019604 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:04.812410 21581 raft_consensus.cc:729] T db24db8916314fdc8d9c8a3b09019604 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:29:04.812708 21581 consensus_queue.cc:260] T db24db8916314fdc8d9c8a3b09019604 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:04.813450 21581 ts_tablet_manager.cc:1398] T db24db8916314fdc8d9c8a3b09019604 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 21:29:04.816457 7236 tablet_copy_service.cc:342] P 39981a8dfe94462b9a73ba489af4c714: Request end of tablet copy session 31a72286397c4edc9a63eca3412be168-db24db8916314fdc8d9c8a3b09019604 received from {username='slave'} at 127.29.82.71:46939
I20240502 21:29:04.816618 7236 tablet_copy_service.cc:433] P 39981a8dfe94462b9a73ba489af4c714: ending tablet copy session 31a72286397c4edc9a63eca3412be168-db24db8916314fdc8d9c8a3b09019604 on tablet db24db8916314fdc8d9c8a3b09019604 with peer 31a72286397c4edc9a63eca3412be168
I20240502 21:29:04.827647 21589 tablet_bootstrap.cc:492] T 9a36933eaf04449faad747e5f44652b9 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:04.827983 21589 tablet_bootstrap.cc:492] T 9a36933eaf04449faad747e5f44652b9 P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap complete.
I20240502 21:29:04.828198 21589 ts_tablet_manager.cc:1367] T 9a36933eaf04449faad747e5f44652b9 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.002s
I20240502 21:29:04.828892 21589 raft_consensus.cc:348] T 9a36933eaf04449faad747e5f44652b9 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:29:04.829201 21589 raft_consensus.cc:729] T 9a36933eaf04449faad747e5f44652b9 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:29:04.829403 21589 consensus_queue.cc:260] T 9a36933eaf04449faad747e5f44652b9 P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:29:04.829986 21589 ts_tablet_manager.cc:1398] T 9a36933eaf04449faad747e5f44652b9 P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:29:04.830744 7370 tablet_copy_service.cc:342] P d0456c5e55a542128688904e003d403c: Request end of tablet copy session 73a1f2ca1f544de9b0e3b37015f85328-9a36933eaf04449faad747e5f44652b9 received from {username='slave'} at 127.29.82.68:49699
I20240502 21:29:04.830861 7370 tablet_copy_service.cc:433] P d0456c5e55a542128688904e003d403c: ending tablet copy session 73a1f2ca1f544de9b0e3b37015f85328-9a36933eaf04449faad747e5f44652b9 on tablet 9a36933eaf04449faad747e5f44652b9 with peer 73a1f2ca1f544de9b0e3b37015f85328
I20240502 21:29:04.855388 21581 ts_tablet_manager.cc:902] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168: Initiating tablet copy from peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:29:04.890121 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:04.907263 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:04.967367 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:04.973182 21627 ts_tablet_manager.cc:902] T 25be332e341d4869b2165da105c17a8a P 398c3c8f3195492c900439ad5e1931bb: Initiating tablet copy from peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:29:04.996475 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:05.069736 21613 ts_tablet_manager.cc:902] T f4cf59f48e874f42aed76d694246e7d3 P 39981a8dfe94462b9a73ba489af4c714: Initiating tablet copy from peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:29:05.070729 21581 tablet_copy_client.cc:308] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.67:34537
I20240502 21:29:05.071591 4727 tablet_copy_service.cc:140] P 398c3c8f3195492c900439ad5e1931bb: Received BeginTabletCopySession request for tablet a4c36adfc86141fbb8724d5a6a798c26 from peer 31a72286397c4edc9a63eca3412be168 ({username='slave'} at 127.29.82.71:44205)
I20240502 21:29:05.071758 4727 tablet_copy_service.cc:161] P 398c3c8f3195492c900439ad5e1931bb: Beginning new tablet copy session on tablet a4c36adfc86141fbb8724d5a6a798c26 from peer 31a72286397c4edc9a63eca3412be168 at {username='slave'} at 127.29.82.71:44205: session id = 31a72286397c4edc9a63eca3412be168-a4c36adfc86141fbb8724d5a6a798c26
W20240502 21:29:05.071813 4375 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.68:42449, user_credentials={real_user=slave}} blocked reactor thread for 136741us
I20240502 21:29:05.074592 4727 tablet_copy_source_session.cc:205] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:05.089660 21581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c36adfc86141fbb8724d5a6a798c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:05.114302 4837 raft_consensus.cc:1203] T 9a36933eaf04449faad747e5f44652b9 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:29:05.126262 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 21:29:05.149649 7549 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.65:42781, user_credentials={real_user=slave}} blocked reactor thread for 72655.5us
I20240502 21:29:05.179299 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:05.203302 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:29:05.210783 21627 tablet_copy_client.cc:308] T 25be332e341d4869b2165da105c17a8a P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.72:41223
I20240502 21:29:05.211548 7638 tablet_copy_service.cc:140] P 0ddd486109f5434cace13d60d633ac04: Received BeginTabletCopySession request for tablet 25be332e341d4869b2165da105c17a8a from peer 398c3c8f3195492c900439ad5e1931bb ({username='slave'} at 127.29.82.67:57193)
I20240502 21:29:05.211763 7638 tablet_copy_service.cc:161] P 0ddd486109f5434cace13d60d633ac04: Beginning new tablet copy session on tablet 25be332e341d4869b2165da105c17a8a from peer 398c3c8f3195492c900439ad5e1931bb at {username='slave'} at 127.29.82.67:57193: session id = 398c3c8f3195492c900439ad5e1931bb-25be332e341d4869b2165da105c17a8a
I20240502 21:29:05.215095 7638 tablet_copy_source_session.cc:205] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:05.216687 21627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25be332e341d4869b2165da105c17a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:05.224213 21627 tablet_copy_client.cc:783] T 25be332e341d4869b2165da105c17a8a P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:05.224645 21627 tablet_copy_client.cc:647] T 25be332e341d4869b2165da105c17a8a P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:05.228075 21627 tablet_copy_client.cc:523] T 25be332e341d4869b2165da105c17a8a P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:05.233960 21627 tablet_bootstrap.cc:492] T 25be332e341d4869b2165da105c17a8a P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:29:05.240404 21627 tablet_bootstrap.cc:492] T 25be332e341d4869b2165da105c17a8a P 398c3c8f3195492c900439ad5e1931bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:05.240739 21627 tablet_bootstrap.cc:492] T 25be332e341d4869b2165da105c17a8a P 398c3c8f3195492c900439ad5e1931bb: Bootstrap complete.
I20240502 21:29:05.240965 21627 ts_tablet_manager.cc:1367] T 25be332e341d4869b2165da105c17a8a P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.001s
I20240502 21:29:05.241832 21627 raft_consensus.cc:348] T 25be332e341d4869b2165da105c17a8a P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:29:05.242031 21627 raft_consensus.cc:729] T 25be332e341d4869b2165da105c17a8a P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:29:05.242415 21627 consensus_queue.cc:260] T 25be332e341d4869b2165da105c17a8a P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:29:05.243139 21627 ts_tablet_manager.cc:1398] T 25be332e341d4869b2165da105c17a8a P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:29:05.244572 7638 tablet_copy_service.cc:342] P 0ddd486109f5434cace13d60d633ac04: Request end of tablet copy session 398c3c8f3195492c900439ad5e1931bb-25be332e341d4869b2165da105c17a8a received from {username='slave'} at 127.29.82.67:57193
I20240502 21:29:05.244714 7638 tablet_copy_service.cc:433] P 0ddd486109f5434cace13d60d633ac04: ending tablet copy session 398c3c8f3195492c900439ad5e1931bb-25be332e341d4869b2165da105c17a8a on tablet 25be332e341d4869b2165da105c17a8a with peer 398c3c8f3195492c900439ad5e1931bb
I20240502 21:29:05.268327 21613 tablet_copy_client.cc:308] T f4cf59f48e874f42aed76d694246e7d3 P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.66:35781
I20240502 21:29:05.269024 4595 tablet_copy_service.cc:140] P 7a736a8ebb164ce0a051c0fccd76c3b8: Received BeginTabletCopySession request for tablet f4cf59f48e874f42aed76d694246e7d3 from peer 39981a8dfe94462b9a73ba489af4c714 ({username='slave'} at 127.29.82.69:37791)
I20240502 21:29:05.269171 4595 tablet_copy_service.cc:161] P 7a736a8ebb164ce0a051c0fccd76c3b8: Beginning new tablet copy session on tablet f4cf59f48e874f42aed76d694246e7d3 from peer 39981a8dfe94462b9a73ba489af4c714 at {username='slave'} at 127.29.82.69:37791: session id = 39981a8dfe94462b9a73ba489af4c714-f4cf59f48e874f42aed76d694246e7d3
I20240502 21:29:05.275053 4595 tablet_copy_source_session.cc:205] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:05.278599 21613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4cf59f48e874f42aed76d694246e7d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:05.288291 21581 tablet_copy_client.cc:783] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:05.288619 21581 tablet_copy_client.cc:647] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:05.292951 21581 tablet_copy_client.cc:523] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:05.305335 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:29:05.308454 21581 tablet_bootstrap.cc:492] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:29:05.310787 7484 raft_consensus.cc:1203] T db24db8916314fdc8d9c8a3b09019604 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:29:05.320295 21613 tablet_copy_client.cc:783] T f4cf59f48e874f42aed76d694246e7d3 P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:05.320672 21613 tablet_copy_client.cc:647] T f4cf59f48e874f42aed76d694246e7d3 P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:05.334456 21613 tablet_copy_client.cc:523] T f4cf59f48e874f42aed76d694246e7d3 P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:05.338451 21613 tablet_bootstrap.cc:492] T f4cf59f48e874f42aed76d694246e7d3 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:29:05.376322 21613 tablet_bootstrap.cc:492] T f4cf59f48e874f42aed76d694246e7d3 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:05.376827 21613 tablet_bootstrap.cc:492] T f4cf59f48e874f42aed76d694246e7d3 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap complete.
I20240502 21:29:05.377111 21613 ts_tablet_manager.cc:1367] T f4cf59f48e874f42aed76d694246e7d3 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.039s user 0.010s sys 0.000s
I20240502 21:29:05.377861 21613 raft_consensus.cc:348] T f4cf59f48e874f42aed76d694246e7d3 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:05.378105 21613 raft_consensus.cc:729] T f4cf59f48e874f42aed76d694246e7d3 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:29:05.378393 21613 consensus_queue.cc:260] T f4cf59f48e874f42aed76d694246e7d3 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:05.383167 21613 ts_tablet_manager.cc:1398] T f4cf59f48e874f42aed76d694246e7d3 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 21:29:05.384543 4595 tablet_copy_service.cc:342] P 7a736a8ebb164ce0a051c0fccd76c3b8: Request end of tablet copy session 39981a8dfe94462b9a73ba489af4c714-f4cf59f48e874f42aed76d694246e7d3 received from {username='slave'} at 127.29.82.69:37791
I20240502 21:29:05.384696 4595 tablet_copy_service.cc:433] P 7a736a8ebb164ce0a051c0fccd76c3b8: ending tablet copy session 39981a8dfe94462b9a73ba489af4c714-f4cf59f48e874f42aed76d694246e7d3 on tablet f4cf59f48e874f42aed76d694246e7d3 with peer 39981a8dfe94462b9a73ba489af4c714
I20240502 21:29:05.389778 21581 tablet_bootstrap.cc:492] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:05.390200 21581 tablet_bootstrap.cc:492] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168: Bootstrap complete.
I20240502 21:29:05.390472 21581 ts_tablet_manager.cc:1367] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.082s user 0.025s sys 0.037s
I20240502 21:29:05.391317 21581 raft_consensus.cc:348] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:05.391585 21581 raft_consensus.cc:729] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:29:05.391954 21581 consensus_queue.cc:260] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:05.392524 21581 ts_tablet_manager.cc:1398] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:29:05.394551 4727 tablet_copy_service.cc:342] P 398c3c8f3195492c900439ad5e1931bb: Request end of tablet copy session 31a72286397c4edc9a63eca3412be168-a4c36adfc86141fbb8724d5a6a798c26 received from {username='slave'} at 127.29.82.71:44205
I20240502 21:29:05.394690 4727 tablet_copy_service.cc:433] P 398c3c8f3195492c900439ad5e1931bb: ending tablet copy session 31a72286397c4edc9a63eca3412be168-a4c36adfc86141fbb8724d5a6a798c26 on tablet a4c36adfc86141fbb8724d5a6a798c26 with peer 31a72286397c4edc9a63eca3412be168
I20240502 21:29:05.424899 4705 raft_consensus.cc:1203] T 25be332e341d4869b2165da105c17a8a P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:29:05.438141 7213 raft_consensus.cc:1203] T f4cf59f48e874f42aed76d694246e7d3 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:29:05.783833 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:05.789362 7483 raft_consensus.cc:1203] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:29:05.814765 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:05.837059 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:05.866830 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:29:05.872293 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:06.004361 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:06.013777 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:06.031862 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:29:06.189540 4704 consensus_queue.cc:237] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:06.191490 7484 raft_consensus.cc:1261] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:29:06.191519 7617 raft_consensus.cc:1261] T a4c36adfc86141fbb8724d5a6a798c26 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:29:06.192050 21516 consensus_queue.cc:1035] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:06.192242 20876 consensus_queue.cc:1035] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:06.194525 21516 raft_consensus.cc:2932] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:06.195916 7617 raft_consensus.cc:2932] T a4c36adfc86141fbb8724d5a6a798c26 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:06.199697 7484 raft_consensus.cc:2932] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:06.201381 4291 catalog_manager.cc:5484] T a4c36adfc86141fbb8724d5a6a798c26 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: config changed from index 2 to 3, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:06.214952 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet a4c36adfc86141fbb8724d5a6a798c26 with delete_type TABLET_DATA_TOMBSTONED (TS 7a736a8ebb164ce0a051c0fccd76c3b8 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:60018
I20240502 21:29:06.215611 21743 tablet_replica.cc:330] T a4c36adfc86141fbb8724d5a6a798c26 P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:29:06.215884 21743 raft_consensus.cc:2229] T a4c36adfc86141fbb8724d5a6a798c26 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:06.216142 21743 raft_consensus.cc:2258] T a4c36adfc86141fbb8724d5a6a798c26 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:06.216864 21743 ts_tablet_manager.cc:1875] T a4c36adfc86141fbb8724d5a6a798c26 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:06.224249 21743 ts_tablet_manager.cc:1888] T a4c36adfc86141fbb8724d5a6a798c26 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:29:06.224398 21743 log.cc:1196] T a4c36adfc86141fbb8724d5a6a798c26 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/a4c36adfc86141fbb8724d5a6a798c26
I20240502 21:29:06.225107 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet a4c36adfc86141fbb8724d5a6a798c26 (table rebalancer_extra_table_42 [id=9c0b534f9084406c81e37e510fcd71d7]) successfully deleted
I20240502 21:29:06.603413 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:06.607445 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:06.621734 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:06.640988 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:06.675956 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:29:06.766798 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:06.809578 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:29:06.848222 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:06.963522 7347 consensus_queue.cc:237] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:29:06.965893 7483 raft_consensus.cc:1261] T 9a36933eaf04449faad747e5f44652b9 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:29:06.965894 4837 raft_consensus.cc:1261] T 9a36933eaf04449faad747e5f44652b9 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:29:06.966385 21293 consensus_queue.cc:1035] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:06.966671 21807 consensus_queue.cc:1035] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:06.969990 21479 raft_consensus.cc:2932] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:29:06.970636 4837 raft_consensus.cc:2932] T 9a36933eaf04449faad747e5f44652b9 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:29:06.970661 7483 raft_consensus.cc:2932] T 9a36933eaf04449faad747e5f44652b9 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:29:06.977763 4300 catalog_manager.cc:5484] T 9a36933eaf04449faad747e5f44652b9 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:29:06.995530 7597 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a36933eaf04449faad747e5f44652b9 with delete_type TABLET_DATA_TOMBSTONED (TS 0ddd486109f5434cace13d60d633ac04 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44444
I20240502 21:29:06.996337 21836 tablet_replica.cc:330] T 9a36933eaf04449faad747e5f44652b9 P 0ddd486109f5434cace13d60d633ac04: stopping tablet replica
I20240502 21:29:06.996668 21836 raft_consensus.cc:2229] T 9a36933eaf04449faad747e5f44652b9 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:06.996966 21836 raft_consensus.cc:2258] T 9a36933eaf04449faad747e5f44652b9 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:06.997560 21836 ts_tablet_manager.cc:1875] T 9a36933eaf04449faad747e5f44652b9 P 0ddd486109f5434cace13d60d633ac04: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:07.004578 21836 ts_tablet_manager.cc:1888] T 9a36933eaf04449faad747e5f44652b9 P 0ddd486109f5434cace13d60d633ac04: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:29:07.004748 21836 log.cc:1196] T 9a36933eaf04449faad747e5f44652b9 P 0ddd486109f5434cace13d60d633ac04: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/wals/9a36933eaf04449faad747e5f44652b9
I20240502 21:29:07.005515 4285 catalog_manager.cc:4830] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): tablet 9a36933eaf04449faad747e5f44652b9 (table rebalancer_extra_table_10 [id=d5866bdd67f64fe5bbeebaed4bd1f67e]) successfully deleted
I20240502 21:29:07.349454 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:07.375347 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:29:07.388414 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:07.458953 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:29:07.494320 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:07.500092 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:07.518779 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:07.572134 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:07.707135 7213 consensus_queue.cc:237] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:07.709430 7483 raft_consensus.cc:1261] T db24db8916314fdc8d9c8a3b09019604 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:29:07.709451 7349 raft_consensus.cc:1261] T db24db8916314fdc8d9c8a3b09019604 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:29:07.709951 21750 consensus_queue.cc:1035] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:07.710510 21750 consensus_queue.cc:1035] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:07.713723 21750 raft_consensus.cc:2932] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:07.714329 7483 raft_consensus.cc:2932] T db24db8916314fdc8d9c8a3b09019604 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:07.720175 4293 catalog_manager.cc:5484] T db24db8916314fdc8d9c8a3b09019604 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: config changed from index 2 to 3, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:07.723531 7349 raft_consensus.cc:2932] T db24db8916314fdc8d9c8a3b09019604 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:07.740490 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet db24db8916314fdc8d9c8a3b09019604 with delete_type TABLET_DATA_TOMBSTONED (TS 7a736a8ebb164ce0a051c0fccd76c3b8 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:60018
I20240502 21:29:07.742856 21935 tablet_replica.cc:330] T db24db8916314fdc8d9c8a3b09019604 P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:29:07.743553 21935 raft_consensus.cc:2229] T db24db8916314fdc8d9c8a3b09019604 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:07.743928 21935 raft_consensus.cc:2258] T db24db8916314fdc8d9c8a3b09019604 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:07.744552 21935 ts_tablet_manager.cc:1875] T db24db8916314fdc8d9c8a3b09019604 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:07.757370 21935 ts_tablet_manager.cc:1888] T db24db8916314fdc8d9c8a3b09019604 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:29:07.757539 21935 log.cc:1196] T db24db8916314fdc8d9c8a3b09019604 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/db24db8916314fdc8d9c8a3b09019604
I20240502 21:29:07.758352 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet db24db8916314fdc8d9c8a3b09019604 (table rebalancer_extra_table_54 [id=ec5b364156ae4e6ebd1fdaca42b585f6]) successfully deleted
I20240502 21:29:08.131675 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:08.148974 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:08.179539 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:29:08.225525 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:08.232573 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:08.285902 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:08.339257 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:08.435194 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:29:08.532860 4573 consensus_queue.cc:237] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:08.534826 4837 raft_consensus.cc:1261] T f4cf59f48e874f42aed76d694246e7d3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:29:08.535388 7213 raft_consensus.cc:1261] T f4cf59f48e874f42aed76d694246e7d3 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:29:08.535854 21831 consensus_queue.cc:1035] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:08.536698 21831 consensus_queue.cc:1035] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:08.538990 21905 raft_consensus.cc:2932] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:08.539822 4837 raft_consensus.cc:2932] T f4cf59f48e874f42aed76d694246e7d3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:08.540771 7213 raft_consensus.cc:2932] T f4cf59f48e874f42aed76d694246e7d3 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:08.546808 4299 catalog_manager.cc:5484] T f4cf59f48e874f42aed76d694246e7d3 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: config changed from index 2 to 3, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:08.571626 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet f4cf59f48e874f42aed76d694246e7d3 with delete_type TABLET_DATA_TOMBSTONED (TS 398c3c8f3195492c900439ad5e1931bb not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44764
I20240502 21:29:08.573325 22061 tablet_replica.cc:330] T f4cf59f48e874f42aed76d694246e7d3 P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:29:08.573561 22061 raft_consensus.cc:2229] T f4cf59f48e874f42aed76d694246e7d3 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:08.573892 22061 raft_consensus.cc:2258] T f4cf59f48e874f42aed76d694246e7d3 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:08.574615 22061 ts_tablet_manager.cc:1875] T f4cf59f48e874f42aed76d694246e7d3 P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:08.579901 22061 ts_tablet_manager.cc:1888] T f4cf59f48e874f42aed76d694246e7d3 P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:29:08.579995 22061 log.cc:1196] T f4cf59f48e874f42aed76d694246e7d3 P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/f4cf59f48e874f42aed76d694246e7d3
I20240502 21:29:08.580696 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet f4cf59f48e874f42aed76d694246e7d3 (table rebalancer_extra_table_42 [id=9c0b534f9084406c81e37e510fcd71d7]) successfully deleted
I20240502 21:29:08.633404 4837 consensus_queue.cc:237] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:08.635757 7349 raft_consensus.cc:1261] T 648263320b474a32bfffa18b1d4a61e3 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:29:08.635867 7483 raft_consensus.cc:1261] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:29:08.636270 22030 consensus_queue.cc:1035] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:29:08.636539 21664 consensus_queue.cc:1035] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:29:08.636878 4771 consensus_peers.cc:485] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): Couldn't send request to peer 0ddd486109f5434cace13d60d633ac04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 648263320b474a32bfffa18b1d4a61e3. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:08.639447 22030 raft_consensus.cc:2932] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:29:08.640026 7483 raft_consensus.cc:2932] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:29:08.648052 4296 catalog_manager.cc:5484] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index -1 to 2, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:29:08.655339 7349 raft_consensus.cc:2932] T 648263320b474a32bfffa18b1d4a61e3 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:29:08.712472 4573 consensus_queue.cc:237] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:08.714906 4837 raft_consensus.cc:1261] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:08.714922 4704 raft_consensus.cc:1261] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:29:08.715026 4510 consensus_peers.cc:485] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8 -> Peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Couldn't send request to peer 39981a8dfe94462b9a73ba489af4c714. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4dc27c0c9cee447f84e4bc1a63f64b68. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:08.715646 22052 consensus_queue.cc:1035] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:08.715948 21905 consensus_queue.cc:1035] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:08.720857 22052 raft_consensus.cc:2932] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:08.721393 4837 raft_consensus.cc:2932] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:08.724603 4704 raft_consensus.cc:2932] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:08.729229 4295 catalog_manager.cc:5484] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index -1 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:08.767966 4703 consensus_queue.cc:237] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:08.781564 4441 raft_consensus.cc:1261] T 128dc7838caa403282ed2868444be271 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:08.782271 21516 consensus_queue.cc:1035] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:29:08.783214 4639 consensus_peers.cc:485] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb -> Peer 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): Couldn't send request to peer 31a72286397c4edc9a63eca3412be168. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 128dc7838caa403282ed2868444be271. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:08.783859 4573 raft_consensus.cc:1261] T 128dc7838caa403282ed2868444be271 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:08.784487 21516 consensus_queue.cc:1035] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:08.806910 21516 raft_consensus.cc:2932] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:08.807780 4441 raft_consensus.cc:2932] T 128dc7838caa403282ed2868444be271 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:08.812726 4573 raft_consensus.cc:2932] T 128dc7838caa403282ed2868444be271 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:08.817842 4295 catalog_manager.cc:5484] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: config changed from index -1 to 4, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:08.901536 4837 consensus_queue.cc:237] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:29:08.904397 7349 raft_consensus.cc:1261] T ed822833ab3c451f96d722980b24c2b0 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:29:08.904641 7213 raft_consensus.cc:1261] T ed822833ab3c451f96d722980b24c2b0 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:29:08.904855 4773 consensus_peers.cc:485] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed822833ab3c451f96d722980b24c2b0. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:08.905117 21838 consensus_queue.cc:1035] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:29:08.905376 22030 consensus_queue.cc:1035] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:29:08.909827 22030 raft_consensus.cc:2932] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:29:08.910427 7349 raft_consensus.cc:2932] T ed822833ab3c451f96d722980b24c2b0 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:29:08.916501 7214 raft_consensus.cc:2932] T ed822833ab3c451f96d722980b24c2b0 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:29:08.922610 4296 catalog_manager.cc:5484] T ed822833ab3c451f96d722980b24c2b0 P d0456c5e55a542128688904e003d403c reported cstate change: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:29:08.964460 7349 consensus_queue.cc:237] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:08.966913 7616 raft_consensus.cc:1261] T 1d3528b68bc740ce9b452356fd26bfb6 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:29:08.967392 7283 consensus_peers.cc:485] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c -> Peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Couldn't send request to peer 39981a8dfe94462b9a73ba489af4c714. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d3528b68bc740ce9b452356fd26bfb6. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:08.967379 4573 raft_consensus.cc:1261] T 1d3528b68bc740ce9b452356fd26bfb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:29:08.967882 22007 consensus_queue.cc:1035] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:29:08.968183 21829 consensus_queue.cc:1035] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:29:08.971230 21829 raft_consensus.cc:2932] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:08.971781 7616 raft_consensus.cc:2932] T 1d3528b68bc740ce9b452356fd26bfb6 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:08.975929 4573 raft_consensus.cc:2932] T 1d3528b68bc740ce9b452356fd26bfb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:08.986449 4295 catalog_manager.cc:5484] T 1d3528b68bc740ce9b452356fd26bfb6 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: config changed from index -1 to 2, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:09.037165 4704 consensus_queue.cc:237] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:29:09.039359 7347 raft_consensus.cc:1261] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:29:09.039824 4642 consensus_peers.cc:485] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3394f5f087304842ada4212f2e524356. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:09.040004 21516 consensus_queue.cc:1035] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:29:09.040411 7616 raft_consensus.cc:1261] T 3394f5f087304842ada4212f2e524356 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:29:09.042037 21516 consensus_queue.cc:1035] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:29:09.060936 22024 raft_consensus.cc:2932] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:29:09.061581 7616 raft_consensus.cc:2932] T 3394f5f087304842ada4212f2e524356 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:29:09.068549 7349 raft_consensus.cc:2932] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:29:09.068712 4296 catalog_manager.cc:5484] T 3394f5f087304842ada4212f2e524356 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: config changed from index -1 to 2, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) added. New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:29:09.126500 7616 consensus_queue.cc:237] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:29:09.129137 7349 raft_consensus.cc:1261] T 6d0334be884c45c3bd9e49095454dfb1 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:29:09.129357 7483 raft_consensus.cc:1261] T 6d0334be884c45c3bd9e49095454dfb1 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:29:09.130244 7552 consensus_peers.cc:485] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d0334be884c45c3bd9e49095454dfb1. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:09.130299 21425 consensus_queue.cc:1035] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:29:09.130571 21936 consensus_queue.cc:1035] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:29:09.152873 21425 raft_consensus.cc:2932] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:29:09.160518 7482 raft_consensus.cc:2932] T 6d0334be884c45c3bd9e49095454dfb1 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:29:09.161422 7349 raft_consensus.cc:2932] T 6d0334be884c45c3bd9e49095454dfb1 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:29:09.166388 4296 catalog_manager.cc:5484] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: config changed from index -1 to 2, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:29:09.214978 7213 consensus_queue.cc:237] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:09.218009 4441 raft_consensus.cc:1261] T 2fd2328258cf4e2db0808731a86cfcf9 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:29:09.218470 7149 consensus_peers.cc:485] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 -> Peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): Couldn't send request to peer 0ddd486109f5434cace13d60d633ac04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fd2328258cf4e2db0808731a86cfcf9. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:09.218839 4704 raft_consensus.cc:1261] T 2fd2328258cf4e2db0808731a86cfcf9 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:29:09.219255 21750 consensus_queue.cc:1035] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:29:09.219543 21600 consensus_queue.cc:1035] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:29:09.222323 22229 ts_tablet_manager.cc:902] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 39981a8dfe94462b9a73ba489af4c714: Initiating tablet copy from peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:29:09.229904 22229 tablet_copy_client.cc:308] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.66:35781
I20240502 21:29:09.229852 21600 raft_consensus.cc:2932] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:29:09.230604 4595 tablet_copy_service.cc:140] P 7a736a8ebb164ce0a051c0fccd76c3b8: Received BeginTabletCopySession request for tablet 4dc27c0c9cee447f84e4bc1a63f64b68 from peer 39981a8dfe94462b9a73ba489af4c714 ({username='slave'} at 127.29.82.69:37791)
I20240502 21:29:09.230767 4595 tablet_copy_service.cc:161] P 7a736a8ebb164ce0a051c0fccd76c3b8: Beginning new tablet copy session on tablet 4dc27c0c9cee447f84e4bc1a63f64b68 from peer 39981a8dfe94462b9a73ba489af4c714 at {username='slave'} at 127.29.82.69:37791: session id = 39981a8dfe94462b9a73ba489af4c714-4dc27c0c9cee447f84e4bc1a63f64b68
I20240502 21:29:09.230543 4441 raft_consensus.cc:2932] T 2fd2328258cf4e2db0808731a86cfcf9 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:29:09.233170 4595 tablet_copy_source_session.cc:205] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:29:09.235045 22229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc27c0c9cee447f84e4bc1a63f64b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:09.235112 4704 raft_consensus.cc:2932] T 2fd2328258cf4e2db0808731a86cfcf9 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:29:09.237895 4299 catalog_manager.cc:5484] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: config changed from index -1 to 2, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:29:09.249367 22229 tablet_copy_client.cc:783] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Starting download of 5 data blocks...
I20240502 21:29:09.267524 22229 tablet_copy_client.cc:647] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:09.269659 22229 tablet_copy_client.cc:523] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:09.273627 22229 tablet_bootstrap.cc:492] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:29:09.275976 22237 ts_tablet_manager.cc:902] T 648263320b474a32bfffa18b1d4a61e3 P 0ddd486109f5434cace13d60d633ac04: Initiating tablet copy from peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:29:09.305583 22229 tablet_bootstrap.cc:492] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:09.306114 22229 tablet_bootstrap.cc:492] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap complete.
I20240502 21:29:09.306447 22229 ts_tablet_manager.cc:1367] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.000s
I20240502 21:29:09.307924 22229 raft_consensus.cc:348] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:09.308200 22229 raft_consensus.cc:729] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:29:09.308462 22229 consensus_queue.cc:260] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:09.309038 22229 ts_tablet_manager.cc:1398] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:29:09.310766 4595 tablet_copy_service.cc:342] P 7a736a8ebb164ce0a051c0fccd76c3b8: Request end of tablet copy session 39981a8dfe94462b9a73ba489af4c714-4dc27c0c9cee447f84e4bc1a63f64b68 received from {username='slave'} at 127.29.82.69:37791
I20240502 21:29:09.310906 4595 tablet_copy_service.cc:433] P 7a736a8ebb164ce0a051c0fccd76c3b8: ending tablet copy session 39981a8dfe94462b9a73ba489af4c714-4dc27c0c9cee447f84e4bc1a63f64b68 on tablet 4dc27c0c9cee447f84e4bc1a63f64b68 with peer 39981a8dfe94462b9a73ba489af4c714
I20240502 21:29:09.317281 22237 tablet_copy_client.cc:308] T 648263320b474a32bfffa18b1d4a61e3 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.68:42449
I20240502 21:29:09.318135 4859 tablet_copy_service.cc:140] P 73a1f2ca1f544de9b0e3b37015f85328: Received BeginTabletCopySession request for tablet 648263320b474a32bfffa18b1d4a61e3 from peer 0ddd486109f5434cace13d60d633ac04 ({username='slave'} at 127.29.82.72:48273)
I20240502 21:29:09.318316 4859 tablet_copy_service.cc:161] P 73a1f2ca1f544de9b0e3b37015f85328: Beginning new tablet copy session on tablet 648263320b474a32bfffa18b1d4a61e3 from peer 0ddd486109f5434cace13d60d633ac04 at {username='slave'} at 127.29.82.72:48273: session id = 0ddd486109f5434cace13d60d633ac04-648263320b474a32bfffa18b1d4a61e3
I20240502 21:29:09.321187 4859 tablet_copy_source_session.cc:205] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:09.323001 22237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 648263320b474a32bfffa18b1d4a61e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:09.329504 22237 tablet_copy_client.cc:783] T 648263320b474a32bfffa18b1d4a61e3 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:09.329726 22237 tablet_copy_client.cc:647] T 648263320b474a32bfffa18b1d4a61e3 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:09.331291 22237 tablet_copy_client.cc:523] T 648263320b474a32bfffa18b1d4a61e3 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:09.333822 22237 tablet_bootstrap.cc:492] T 648263320b474a32bfffa18b1d4a61e3 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:29:09.339170 22237 tablet_bootstrap.cc:492] T 648263320b474a32bfffa18b1d4a61e3 P 0ddd486109f5434cace13d60d633ac04: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:09.339602 22237 tablet_bootstrap.cc:492] T 648263320b474a32bfffa18b1d4a61e3 P 0ddd486109f5434cace13d60d633ac04: Bootstrap complete.
I20240502 21:29:09.339836 22237 ts_tablet_manager.cc:1367] T 648263320b474a32bfffa18b1d4a61e3 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.003s
I20240502 21:29:09.340461 22237 raft_consensus.cc:348] T 648263320b474a32bfffa18b1d4a61e3 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:09.340757 22237 raft_consensus.cc:729] T 648263320b474a32bfffa18b1d4a61e3 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:29:09.340961 22237 consensus_queue.cc:260] T 648263320b474a32bfffa18b1d4a61e3 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:09.341598 22237 ts_tablet_manager.cc:1398] T 648263320b474a32bfffa18b1d4a61e3 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:29:09.356225 4859 tablet_copy_service.cc:342] P 73a1f2ca1f544de9b0e3b37015f85328: Request end of tablet copy session 0ddd486109f5434cace13d60d633ac04-648263320b474a32bfffa18b1d4a61e3 received from {username='slave'} at 127.29.82.72:48273
I20240502 21:29:09.356866 4859 tablet_copy_service.cc:433] P 73a1f2ca1f544de9b0e3b37015f85328: ending tablet copy session 0ddd486109f5434cace13d60d633ac04-648263320b474a32bfffa18b1d4a61e3 on tablet 648263320b474a32bfffa18b1d4a61e3 with peer 0ddd486109f5434cace13d60d633ac04
I20240502 21:29:09.374425 22255 ts_tablet_manager.cc:902] T 128dc7838caa403282ed2868444be271 P 31a72286397c4edc9a63eca3412be168: Initiating tablet copy from peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:29:09.397429 22258 ts_tablet_manager.cc:902] T ed822833ab3c451f96d722980b24c2b0 P 398c3c8f3195492c900439ad5e1931bb: Initiating tablet copy from peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:29:09.434188 22255 tablet_copy_client.cc:308] T 128dc7838caa403282ed2868444be271 P 31a72286397c4edc9a63eca3412be168: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.67:34537
I20240502 21:29:09.434849 4727 tablet_copy_service.cc:140] P 398c3c8f3195492c900439ad5e1931bb: Received BeginTabletCopySession request for tablet 128dc7838caa403282ed2868444be271 from peer 31a72286397c4edc9a63eca3412be168 ({username='slave'} at 127.29.82.71:44205)
I20240502 21:29:09.435014 4727 tablet_copy_service.cc:161] P 398c3c8f3195492c900439ad5e1931bb: Beginning new tablet copy session on tablet 128dc7838caa403282ed2868444be271 from peer 31a72286397c4edc9a63eca3412be168 at {username='slave'} at 127.29.82.71:44205: session id = 31a72286397c4edc9a63eca3412be168-128dc7838caa403282ed2868444be271
I20240502 21:29:09.438383 4727 tablet_copy_source_session.cc:205] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:29:09.439916 22255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 128dc7838caa403282ed2868444be271. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:09.446666 22255 tablet_copy_client.cc:783] T 128dc7838caa403282ed2868444be271 P 31a72286397c4edc9a63eca3412be168: tablet copy: Starting download of 5 data blocks...
I20240502 21:29:09.451750 22258 tablet_copy_client.cc:308] T ed822833ab3c451f96d722980b24c2b0 P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.68:42449
I20240502 21:29:09.452489 4859 tablet_copy_service.cc:140] P 73a1f2ca1f544de9b0e3b37015f85328: Received BeginTabletCopySession request for tablet ed822833ab3c451f96d722980b24c2b0 from peer 398c3c8f3195492c900439ad5e1931bb ({username='slave'} at 127.29.82.67:42171)
I20240502 21:29:09.452682 4859 tablet_copy_service.cc:161] P 73a1f2ca1f544de9b0e3b37015f85328: Beginning new tablet copy session on tablet ed822833ab3c451f96d722980b24c2b0 from peer 398c3c8f3195492c900439ad5e1931bb at {username='slave'} at 127.29.82.67:42171: session id = 398c3c8f3195492c900439ad5e1931bb-ed822833ab3c451f96d722980b24c2b0
I20240502 21:29:09.463025 4859 tablet_copy_source_session.cc:205] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:09.464373 22258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed822833ab3c451f96d722980b24c2b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:09.467800 22255 tablet_copy_client.cc:647] T 128dc7838caa403282ed2868444be271 P 31a72286397c4edc9a63eca3412be168: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:09.470580 22255 tablet_copy_client.cc:523] T 128dc7838caa403282ed2868444be271 P 31a72286397c4edc9a63eca3412be168: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:09.471653 22258 tablet_copy_client.cc:783] T ed822833ab3c451f96d722980b24c2b0 P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:09.471855 22258 tablet_copy_client.cc:647] T ed822833ab3c451f96d722980b24c2b0 P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:09.475104 22255 tablet_bootstrap.cc:492] T 128dc7838caa403282ed2868444be271 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:29:09.479769 22258 tablet_copy_client.cc:523] T ed822833ab3c451f96d722980b24c2b0 P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:09.481814 22229 ts_tablet_manager.cc:902] T 1d3528b68bc740ce9b452356fd26bfb6 P 39981a8dfe94462b9a73ba489af4c714: Initiating tablet copy from peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:29:09.492535 22258 tablet_bootstrap.cc:492] T ed822833ab3c451f96d722980b24c2b0 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:29:09.498178 22276 ts_tablet_manager.cc:902] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8: Initiating tablet copy from peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:29:09.508646 22229 tablet_copy_client.cc:308] T 1d3528b68bc740ce9b452356fd26bfb6 P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.70:40719
I20240502 21:29:09.509508 7370 tablet_copy_service.cc:140] P d0456c5e55a542128688904e003d403c: Received BeginTabletCopySession request for tablet 1d3528b68bc740ce9b452356fd26bfb6 from peer 39981a8dfe94462b9a73ba489af4c714 ({username='slave'} at 127.29.82.69:58215)
I20240502 21:29:09.509688 7370 tablet_copy_service.cc:161] P d0456c5e55a542128688904e003d403c: Beginning new tablet copy session on tablet 1d3528b68bc740ce9b452356fd26bfb6 from peer 39981a8dfe94462b9a73ba489af4c714 at {username='slave'} at 127.29.82.69:58215: session id = 39981a8dfe94462b9a73ba489af4c714-1d3528b68bc740ce9b452356fd26bfb6
I20240502 21:29:09.511659 22258 tablet_bootstrap.cc:492] T ed822833ab3c451f96d722980b24c2b0 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:09.512077 22258 tablet_bootstrap.cc:492] T ed822833ab3c451f96d722980b24c2b0 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap complete.
I20240502 21:29:09.512367 22258 ts_tablet_manager.cc:1367] T ed822833ab3c451f96d722980b24c2b0 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.000s
I20240502 21:29:09.512678 7370 tablet_copy_source_session.cc:205] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:09.513206 22258 raft_consensus.cc:348] T ed822833ab3c451f96d722980b24c2b0 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:29:09.513438 22258 raft_consensus.cc:729] T ed822833ab3c451f96d722980b24c2b0 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:29:09.513633 22258 consensus_queue.cc:260] T ed822833ab3c451f96d722980b24c2b0 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:29:09.517858 22229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3528b68bc740ce9b452356fd26bfb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:09.533480 22258 ts_tablet_manager.cc:1398] T ed822833ab3c451f96d722980b24c2b0 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.021s user 0.001s sys 0.000s
I20240502 21:29:09.535490 4859 tablet_copy_service.cc:342] P 73a1f2ca1f544de9b0e3b37015f85328: Request end of tablet copy session 398c3c8f3195492c900439ad5e1931bb-ed822833ab3c451f96d722980b24c2b0 received from {username='slave'} at 127.29.82.67:42171
I20240502 21:29:09.535653 22229 tablet_copy_client.cc:783] T 1d3528b68bc740ce9b452356fd26bfb6 P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:09.535677 4859 tablet_copy_service.cc:433] P 73a1f2ca1f544de9b0e3b37015f85328: ending tablet copy session 398c3c8f3195492c900439ad5e1931bb-ed822833ab3c451f96d722980b24c2b0 on tablet ed822833ab3c451f96d722980b24c2b0 with peer 398c3c8f3195492c900439ad5e1931bb
I20240502 21:29:09.535903 22229 tablet_copy_client.cc:647] T 1d3528b68bc740ce9b452356fd26bfb6 P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:09.537432 22229 tablet_copy_client.cc:523] T 1d3528b68bc740ce9b452356fd26bfb6 P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:09.540881 22229 tablet_bootstrap.cc:492] T 1d3528b68bc740ce9b452356fd26bfb6 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:29:09.546584 22229 tablet_bootstrap.cc:492] T 1d3528b68bc740ce9b452356fd26bfb6 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:09.546947 22229 tablet_bootstrap.cc:492] T 1d3528b68bc740ce9b452356fd26bfb6 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap complete.
I20240502 21:29:09.547205 22229 ts_tablet_manager.cc:1367] T 1d3528b68bc740ce9b452356fd26bfb6 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20240502 21:29:09.547883 22229 raft_consensus.cc:348] T 1d3528b68bc740ce9b452356fd26bfb6 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:09.548132 22229 raft_consensus.cc:729] T 1d3528b68bc740ce9b452356fd26bfb6 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:29:09.548426 22229 consensus_queue.cc:260] T 1d3528b68bc740ce9b452356fd26bfb6 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:09.549152 22229 ts_tablet_manager.cc:1398] T 1d3528b68bc740ce9b452356fd26bfb6 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:29:09.551330 7370 tablet_copy_service.cc:342] P d0456c5e55a542128688904e003d403c: Request end of tablet copy session 39981a8dfe94462b9a73ba489af4c714-1d3528b68bc740ce9b452356fd26bfb6 received from {username='slave'} at 127.29.82.69:58215
I20240502 21:29:09.551462 7370 tablet_copy_service.cc:433] P d0456c5e55a542128688904e003d403c: ending tablet copy session 39981a8dfe94462b9a73ba489af4c714-1d3528b68bc740ce9b452356fd26bfb6 on tablet 1d3528b68bc740ce9b452356fd26bfb6 with peer 39981a8dfe94462b9a73ba489af4c714
I20240502 21:29:09.557329 7262 maintenance_manager.cc:419] P 39981a8dfe94462b9a73ba489af4c714: Scheduling UndoDeltaBlockGCOp(4dc27c0c9cee447f84e4bc1a63f64b68): 4586 bytes on disk
I20240502 21:29:09.557931 7152 maintenance_manager.cc:643] P 39981a8dfe94462b9a73ba489af4c714: UndoDeltaBlockGCOp(4dc27c0c9cee447f84e4bc1a63f64b68) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240502 21:29:09.561285 22255 tablet_bootstrap.cc:492] T 128dc7838caa403282ed2868444be271 P 31a72286397c4edc9a63eca3412be168: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:09.561846 22255 tablet_bootstrap.cc:492] T 128dc7838caa403282ed2868444be271 P 31a72286397c4edc9a63eca3412be168: Bootstrap complete.
I20240502 21:29:09.562307 22255 ts_tablet_manager.cc:1367] T 128dc7838caa403282ed2868444be271 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.087s user 0.024s sys 0.008s
I20240502 21:29:09.562999 22276 tablet_copy_client.cc:308] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.67:34537
I20240502 21:29:09.563160 22255 raft_consensus.cc:348] T 128dc7838caa403282ed2868444be271 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:09.563477 22255 raft_consensus.cc:729] T 128dc7838caa403282ed2868444be271 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:29:09.563688 4727 tablet_copy_service.cc:140] P 398c3c8f3195492c900439ad5e1931bb: Received BeginTabletCopySession request for tablet 3394f5f087304842ada4212f2e524356 from peer 7a736a8ebb164ce0a051c0fccd76c3b8 ({username='slave'} at 127.29.82.66:50289)
I20240502 21:29:09.563822 4727 tablet_copy_service.cc:161] P 398c3c8f3195492c900439ad5e1931bb: Beginning new tablet copy session on tablet 3394f5f087304842ada4212f2e524356 from peer 7a736a8ebb164ce0a051c0fccd76c3b8 at {username='slave'} at 127.29.82.66:50289: session id = 7a736a8ebb164ce0a051c0fccd76c3b8-3394f5f087304842ada4212f2e524356
I20240502 21:29:09.563761 22255 consensus_queue.cc:260] T 128dc7838caa403282ed2868444be271 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:09.566522 22255 ts_tablet_manager.cc:1398] T 128dc7838caa403282ed2868444be271 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:29:09.566915 4727 tablet_copy_source_session.cc:205] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:09.567745 4726 tablet_copy_service.cc:342] P 398c3c8f3195492c900439ad5e1931bb: Request end of tablet copy session 31a72286397c4edc9a63eca3412be168-128dc7838caa403282ed2868444be271 received from {username='slave'} at 127.29.82.71:44205
I20240502 21:29:09.567869 4726 tablet_copy_service.cc:433] P 398c3c8f3195492c900439ad5e1931bb: ending tablet copy session 31a72286397c4edc9a63eca3412be168-128dc7838caa403282ed2868444be271 on tablet 128dc7838caa403282ed2868444be271 with peer 31a72286397c4edc9a63eca3412be168
I20240502 21:29:09.568625 22276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3394f5f087304842ada4212f2e524356. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:09.591114 22276 tablet_copy_client.cc:783] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:09.591435 22276 tablet_copy_client.cc:647] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:09.594995 22276 tablet_copy_client.cc:523] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:09.601130 22276 tablet_bootstrap.cc:492] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap starting.
I20240502 21:29:09.624361 22276 tablet_bootstrap.cc:492] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:09.624841 22276 tablet_bootstrap.cc:492] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8: Bootstrap complete.
I20240502 21:29:09.625159 22276 ts_tablet_manager.cc:1367] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.002s
I20240502 21:29:09.626168 22276 raft_consensus.cc:348] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:29:09.626511 22276 raft_consensus.cc:729] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Initialized, Role: FOLLOWER
I20240502 21:29:09.626823 22276 consensus_queue.cc:260] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:29:09.627612 22276 ts_tablet_manager.cc:1398] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:29:09.631911 4726 tablet_copy_service.cc:342] P 398c3c8f3195492c900439ad5e1931bb: Request end of tablet copy session 7a736a8ebb164ce0a051c0fccd76c3b8-3394f5f087304842ada4212f2e524356 received from {username='slave'} at 127.29.82.66:50289
I20240502 21:29:09.632056 4726 tablet_copy_service.cc:433] P 398c3c8f3195492c900439ad5e1931bb: ending tablet copy session 7a736a8ebb164ce0a051c0fccd76c3b8-3394f5f087304842ada4212f2e524356 on tablet 3394f5f087304842ada4212f2e524356 with peer 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:29:09.672619 7531 maintenance_manager.cc:419] P 31a72286397c4edc9a63eca3412be168: Scheduling UndoDeltaBlockGCOp(128dc7838caa403282ed2868444be271): 4925 bytes on disk
I20240502 21:29:09.673739 7214 raft_consensus.cc:1203] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:29:09.674515 7421 maintenance_manager.cc:643] P 31a72286397c4edc9a63eca3412be168: UndoDeltaBlockGCOp(128dc7838caa403282ed2868444be271) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":126,"lbm_reads_lt_1ms":4}
I20240502 21:29:09.688833 22258 ts_tablet_manager.cc:902] T 6d0334be884c45c3bd9e49095454dfb1 P 398c3c8f3195492c900439ad5e1931bb: Initiating tablet copy from peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:29:09.746939 7483 raft_consensus.cc:1203] T 128dc7838caa403282ed2868444be271 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:29:09.803185 22258 tablet_copy_client.cc:308] T 6d0334be884c45c3bd9e49095454dfb1 P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.72:41223
I20240502 21:29:09.804142 7638 tablet_copy_service.cc:140] P 0ddd486109f5434cace13d60d633ac04: Received BeginTabletCopySession request for tablet 6d0334be884c45c3bd9e49095454dfb1 from peer 398c3c8f3195492c900439ad5e1931bb ({username='slave'} at 127.29.82.67:57193)
I20240502 21:29:09.804299 7638 tablet_copy_service.cc:161] P 0ddd486109f5434cace13d60d633ac04: Beginning new tablet copy session on tablet 6d0334be884c45c3bd9e49095454dfb1 from peer 398c3c8f3195492c900439ad5e1931bb at {username='slave'} at 127.29.82.67:57193: session id = 398c3c8f3195492c900439ad5e1931bb-6d0334be884c45c3bd9e49095454dfb1
I20240502 21:29:09.807024 7638 tablet_copy_source_session.cc:205] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:09.810417 22258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d0334be884c45c3bd9e49095454dfb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:09.810945 7616 raft_consensus.cc:1203] T 648263320b474a32bfffa18b1d4a61e3 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:29:09.836040 22237 ts_tablet_manager.cc:902] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04: Initiating tablet copy from peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:29:09.825969 22258 tablet_copy_client.cc:783] T 6d0334be884c45c3bd9e49095454dfb1 P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:09.849556 22258 tablet_copy_client.cc:647] T 6d0334be884c45c3bd9e49095454dfb1 P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:09.852139 22258 tablet_copy_client.cc:523] T 6d0334be884c45c3bd9e49095454dfb1 P 398c3c8f3195492c900439ad5e1931bb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:09.879585 4704 raft_consensus.cc:1203] T ed822833ab3c451f96d722980b24c2b0 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:29:09.886680 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:09.890146 22258 tablet_bootstrap.cc:492] T 6d0334be884c45c3bd9e49095454dfb1 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap starting.
I20240502 21:29:09.898284 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:09.913808 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:09.932595 22258 tablet_bootstrap.cc:492] T 6d0334be884c45c3bd9e49095454dfb1 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:09.933092 22258 tablet_bootstrap.cc:492] T 6d0334be884c45c3bd9e49095454dfb1 P 398c3c8f3195492c900439ad5e1931bb: Bootstrap complete.
I20240502 21:29:09.933410 22258 ts_tablet_manager.cc:1367] T 6d0334be884c45c3bd9e49095454dfb1 P 398c3c8f3195492c900439ad5e1931bb: Time spent bootstrapping tablet: real 0.043s user 0.009s sys 0.000s
I20240502 21:29:09.934252 22258 raft_consensus.cc:348] T 6d0334be884c45c3bd9e49095454dfb1 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:29:09.935137 22258 raft_consensus.cc:729] T 6d0334be884c45c3bd9e49095454dfb1 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Initialized, Role: FOLLOWER
I20240502 21:29:09.935720 22258 consensus_queue.cc:260] T 6d0334be884c45c3bd9e49095454dfb1 P 398c3c8f3195492c900439ad5e1931bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:29:09.950379 22258 ts_tablet_manager.cc:1398] T 6d0334be884c45c3bd9e49095454dfb1 P 398c3c8f3195492c900439ad5e1931bb: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240502 21:29:09.964341 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:29:09.979080 7638 tablet_copy_service.cc:342] P 0ddd486109f5434cace13d60d633ac04: Request end of tablet copy session 398c3c8f3195492c900439ad5e1931bb-6d0334be884c45c3bd9e49095454dfb1 received from {username='slave'} at 127.29.82.67:57193
I20240502 21:29:09.979269 7638 tablet_copy_service.cc:433] P 0ddd486109f5434cace13d60d633ac04: ending tablet copy session 398c3c8f3195492c900439ad5e1931bb-6d0334be884c45c3bd9e49095454dfb1 on tablet 6d0334be884c45c3bd9e49095454dfb1 with peer 398c3c8f3195492c900439ad5e1931bb
I20240502 21:29:09.990772 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:10.001552 4575 raft_consensus.cc:1203] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:29:10.012951 7213 raft_consensus.cc:1203] T 1d3528b68bc740ce9b452356fd26bfb6 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:29:10.033056 22237 tablet_copy_client.cc:308] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.69:33193
I20240502 21:29:10.033682 7236 tablet_copy_service.cc:140] P 39981a8dfe94462b9a73ba489af4c714: Received BeginTabletCopySession request for tablet 2fd2328258cf4e2db0808731a86cfcf9 from peer 0ddd486109f5434cace13d60d633ac04 ({username='slave'} at 127.29.82.72:50223)
I20240502 21:29:10.033843 7236 tablet_copy_service.cc:161] P 39981a8dfe94462b9a73ba489af4c714: Beginning new tablet copy session on tablet 2fd2328258cf4e2db0808731a86cfcf9 from peer 0ddd486109f5434cace13d60d633ac04 at {username='slave'} at 127.29.82.72:50223: session id = 0ddd486109f5434cace13d60d633ac04-2fd2328258cf4e2db0808731a86cfcf9
I20240502 21:29:10.036676 7236 tablet_copy_source_session.cc:205] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:10.038213 22237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fd2328258cf4e2db0808731a86cfcf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:10.071434 22237 tablet_copy_client.cc:783] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:10.072175 22237 tablet_copy_client.cc:647] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:10.076413 22237 tablet_copy_client.cc:523] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:10.092368 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:10.098834 22237 tablet_bootstrap.cc:492] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:29:10.105259 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:10.109738 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:29:10.150113 22237 tablet_bootstrap.cc:492] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:10.150527 22237 tablet_bootstrap.cc:492] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04: Bootstrap complete.
I20240502 21:29:10.150794 22237 ts_tablet_manager.cc:1367] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.052s user 0.007s sys 0.000s
I20240502 21:29:10.151525 22237 raft_consensus.cc:348] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:10.156014 22237 raft_consensus.cc:729] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:29:10.158843 22237 consensus_queue.cc:260] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:10.162742 22237 ts_tablet_manager.cc:1398] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20240502 21:29:10.163467 7236 tablet_copy_service.cc:342] P 39981a8dfe94462b9a73ba489af4c714: Request end of tablet copy session 0ddd486109f5434cace13d60d633ac04-2fd2328258cf4e2db0808731a86cfcf9 received from {username='slave'} at 127.29.82.72:50223
I20240502 21:29:10.163610 7236 tablet_copy_service.cc:433] P 39981a8dfe94462b9a73ba489af4c714: ending tablet copy session 0ddd486109f5434cace13d60d633ac04-2fd2328258cf4e2db0808731a86cfcf9 on tablet 2fd2328258cf4e2db0808731a86cfcf9 with peer 0ddd486109f5434cace13d60d633ac04
I20240502 21:29:10.223821 4704 raft_consensus.cc:1203] T 6d0334be884c45c3bd9e49095454dfb1 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:29:10.456679 7616 raft_consensus.cc:1203] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:29:10.711355 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:10.740710 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:10.750310 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:29:10.783897 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:10.826020 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:10.845628 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:10.926780 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:10.929219 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:29:11.079285 4705 consensus_queue.cc:237] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:29:11.093883 7349 raft_consensus.cc:1261] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:29:11.093957 4575 raft_consensus.cc:1261] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:29:11.094666 22127 consensus_queue.cc:1035] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:11.097306 22428 consensus_queue.cc:1035] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:11.100005 22127 raft_consensus.cc:2932] T 3394f5f087304842ada4212f2e524356 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:29:11.100545 7349 raft_consensus.cc:2932] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:29:11.101081 4575 raft_consensus.cc:2932] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:29:11.107491 4298 catalog_manager.cc:5484] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c reported cstate change: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:29:11.140404 7597 tablet_service.cc:1514] Processing DeleteTablet for tablet 3394f5f087304842ada4212f2e524356 with delete_type TABLET_DATA_TOMBSTONED (TS 0ddd486109f5434cace13d60d633ac04 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44444
I20240502 21:29:11.141186 22442 tablet_replica.cc:330] T 3394f5f087304842ada4212f2e524356 P 0ddd486109f5434cace13d60d633ac04: stopping tablet replica
I20240502 21:29:11.141649 22442 raft_consensus.cc:2229] T 3394f5f087304842ada4212f2e524356 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:11.142015 22442 raft_consensus.cc:2258] T 3394f5f087304842ada4212f2e524356 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:11.143472 22442 ts_tablet_manager.cc:1875] T 3394f5f087304842ada4212f2e524356 P 0ddd486109f5434cace13d60d633ac04: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:11.149682 22442 ts_tablet_manager.cc:1888] T 3394f5f087304842ada4212f2e524356 P 0ddd486109f5434cace13d60d633ac04: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:29:11.149784 22442 log.cc:1196] T 3394f5f087304842ada4212f2e524356 P 0ddd486109f5434cace13d60d633ac04: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/wals/3394f5f087304842ada4212f2e524356
I20240502 21:29:11.150467 4285 catalog_manager.cc:4830] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): tablet 3394f5f087304842ada4212f2e524356 (table rebalancer_extra_table_30 [id=b87f68aedb34492aade9e648ad016b2c]) successfully deleted
I20240502 21:29:11.485955 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:11.534025 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:29:11.545694 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:11.566170 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:11.598754 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:11.626865 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:11.665786 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:29:11.679141 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:11.850199 7350 consensus_queue.cc:237] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:11.852737 7214 raft_consensus.cc:1261] T 1d3528b68bc740ce9b452356fd26bfb6 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:29:11.852753 4573 raft_consensus.cc:1261] T 1d3528b68bc740ce9b452356fd26bfb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:29:11.853300 22504 consensus_queue.cc:1035] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:11.853533 22007 consensus_queue.cc:1035] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:11.856572 22500 raft_consensus.cc:2932] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:11.860275 4575 raft_consensus.cc:2932] T 1d3528b68bc740ce9b452356fd26bfb6 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:11.868116 4295 catalog_manager.cc:5484] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c reported cstate change: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:11.878621 7214 raft_consensus.cc:2932] T 1d3528b68bc740ce9b452356fd26bfb6 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:11.904835 7597 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d3528b68bc740ce9b452356fd26bfb6 with delete_type TABLET_DATA_TOMBSTONED (TS 0ddd486109f5434cace13d60d633ac04 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44444
I20240502 21:29:11.905444 22533 tablet_replica.cc:330] T 1d3528b68bc740ce9b452356fd26bfb6 P 0ddd486109f5434cace13d60d633ac04: stopping tablet replica
I20240502 21:29:11.905730 22533 raft_consensus.cc:2229] T 1d3528b68bc740ce9b452356fd26bfb6 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:11.906315 22533 raft_consensus.cc:2258] T 1d3528b68bc740ce9b452356fd26bfb6 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:11.907048 22533 ts_tablet_manager.cc:1875] T 1d3528b68bc740ce9b452356fd26bfb6 P 0ddd486109f5434cace13d60d633ac04: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:11.914017 22533 ts_tablet_manager.cc:1888] T 1d3528b68bc740ce9b452356fd26bfb6 P 0ddd486109f5434cace13d60d633ac04: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:29:11.914201 22533 log.cc:1196] T 1d3528b68bc740ce9b452356fd26bfb6 P 0ddd486109f5434cace13d60d633ac04: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/wals/1d3528b68bc740ce9b452356fd26bfb6
I20240502 21:29:11.915033 4285 catalog_manager.cc:4830] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): tablet 1d3528b68bc740ce9b452356fd26bfb6 (table rebalancer_extra_table_4 [id=2ba724ea31944444b73e3c52e98f06e7]) successfully deleted
I20240502 21:29:12.271173 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:12.311821 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:12.350323 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:29:12.367117 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:12.414248 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:12.418262 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:12.431838 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:12.457852 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:29:12.675374 4705 consensus_queue.cc:237] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:12.680805 7484 raft_consensus.cc:1261] T 128dc7838caa403282ed2868444be271 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:12.680914 4441 raft_consensus.cc:1261] T 128dc7838caa403282ed2868444be271 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:12.681313 22575 consensus_queue.cc:1035] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:12.681650 21516 consensus_queue.cc:1035] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:12.684731 22575 raft_consensus.cc:2932] T 128dc7838caa403282ed2868444be271 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:12.685302 4441 raft_consensus.cc:2932] T 128dc7838caa403282ed2868444be271 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:12.690440 7482 raft_consensus.cc:2932] T 128dc7838caa403282ed2868444be271 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:12.695537 4295 catalog_manager.cc:5484] T 128dc7838caa403282ed2868444be271 P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: config changed from index 4 to 5, VOTER 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) evicted. New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:12.704646 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet 128dc7838caa403282ed2868444be271 with delete_type TABLET_DATA_TOMBSTONED (TS 7a736a8ebb164ce0a051c0fccd76c3b8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60018
I20240502 21:29:12.705358 22628 tablet_replica.cc:330] T 128dc7838caa403282ed2868444be271 P 7a736a8ebb164ce0a051c0fccd76c3b8: stopping tablet replica
I20240502 21:29:12.705636 22628 raft_consensus.cc:2229] T 128dc7838caa403282ed2868444be271 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:12.706328 22628 raft_consensus.cc:2258] T 128dc7838caa403282ed2868444be271 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:12.707624 22628 ts_tablet_manager.cc:1875] T 128dc7838caa403282ed2868444be271 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:12.722210 22628 ts_tablet_manager.cc:1888] T 128dc7838caa403282ed2868444be271 P 7a736a8ebb164ce0a051c0fccd76c3b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:29:12.722426 22628 log.cc:1196] T 128dc7838caa403282ed2868444be271 P 7a736a8ebb164ce0a051c0fccd76c3b8: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/wals/128dc7838caa403282ed2868444be271
I20240502 21:29:12.727994 4287 catalog_manager.cc:4830] TS 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): tablet 128dc7838caa403282ed2868444be271 (table rebalance_test_table_11 [id=6578c35615d249838e00b267cf4291d8]) successfully deleted
I20240502 21:29:13.097960 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:13.118645 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:13.125866 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:13.139060 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:29:13.199538 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:13.261466 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:13.274645 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:13.296607 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:29:13.444205 4836 consensus_queue.cc:237] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:29:13.446265 7214 raft_consensus.cc:1261] T ed822833ab3c451f96d722980b24c2b0 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:29:13.446818 21838 consensus_queue.cc:1035] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:13.448089 4704 raft_consensus.cc:1261] T ed822833ab3c451f96d722980b24c2b0 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:29:13.448681 21838 consensus_queue.cc:1035] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:13.449599 22030 raft_consensus.cc:2932] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:29:13.450187 7214 raft_consensus.cc:2932] T ed822833ab3c451f96d722980b24c2b0 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:29:13.452702 4704 raft_consensus.cc:2932] T ed822833ab3c451f96d722980b24c2b0 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:29:13.456487 4294 catalog_manager.cc:5484] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:29:13.484328 7328 tablet_service.cc:1514] Processing DeleteTablet for tablet ed822833ab3c451f96d722980b24c2b0 with delete_type TABLET_DATA_TOMBSTONED (TS d0456c5e55a542128688904e003d403c not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47078
I20240502 21:29:13.485065 22726 tablet_replica.cc:330] T ed822833ab3c451f96d722980b24c2b0 P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:29:13.485383 22726 raft_consensus.cc:2229] T ed822833ab3c451f96d722980b24c2b0 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:13.486410 22726 raft_consensus.cc:2258] T ed822833ab3c451f96d722980b24c2b0 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:13.487267 22726 ts_tablet_manager.cc:1875] T ed822833ab3c451f96d722980b24c2b0 P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:13.495510 22726 ts_tablet_manager.cc:1888] T ed822833ab3c451f96d722980b24c2b0 P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:29:13.495752 22726 log.cc:1196] T ed822833ab3c451f96d722980b24c2b0 P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/ed822833ab3c451f96d722980b24c2b0
I20240502 21:29:13.496795 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet ed822833ab3c451f96d722980b24c2b0 (table rebalancer_extra_table_6 [id=6a89f5df3f6a4844a9c775290e276fa0]) successfully deleted
I20240502 21:29:14.013721 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:14.037832 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:14.049454 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:14.089638 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:29:14.089640 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:14.094069 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:14.097386 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:14.107831 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:29:14.309578 4836 consensus_queue.cc:237] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:14.311618 7618 raft_consensus.cc:1261] T 648263320b474a32bfffa18b1d4a61e3 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:29:14.311878 7484 raft_consensus.cc:1261] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:29:14.312285 22030 consensus_queue.cc:1035] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:14.312531 21838 consensus_queue.cc:1035] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:14.329316 21838 raft_consensus.cc:2932] T 648263320b474a32bfffa18b1d4a61e3 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:29:14.329998 7484 raft_consensus.cc:2932] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:29:14.330307 7618 raft_consensus.cc:2932] T 648263320b474a32bfffa18b1d4a61e3 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:29:14.337709 4294 catalog_manager.cc:5484] T 648263320b474a32bfffa18b1d4a61e3 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:29:14.376026 7328 tablet_service.cc:1514] Processing DeleteTablet for tablet 648263320b474a32bfffa18b1d4a61e3 with delete_type TABLET_DATA_TOMBSTONED (TS d0456c5e55a542128688904e003d403c not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47078
I20240502 21:29:14.387465 22827 tablet_replica.cc:330] T 648263320b474a32bfffa18b1d4a61e3 P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:29:14.387773 22827 raft_consensus.cc:2229] T 648263320b474a32bfffa18b1d4a61e3 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:14.388262 22827 raft_consensus.cc:2258] T 648263320b474a32bfffa18b1d4a61e3 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:14.392735 22827 ts_tablet_manager.cc:1875] T 648263320b474a32bfffa18b1d4a61e3 P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:14.408715 22827 ts_tablet_manager.cc:1888] T 648263320b474a32bfffa18b1d4a61e3 P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:29:14.408852 22827 log.cc:1196] T 648263320b474a32bfffa18b1d4a61e3 P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/648263320b474a32bfffa18b1d4a61e3
I20240502 21:29:14.409868 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet 648263320b474a32bfffa18b1d4a61e3 (table rebalancer_extra_table_8 [id=a99691f2b4cb42afb50b56d30958d085]) successfully deleted
I20240502 21:29:14.708068 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:14.719934 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:14.756156 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:14.791105 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:14.792304 7194 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:29:14.804661 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:14.806268 4686 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:14.889631 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:29:14.998543 7213 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2fd2328258cf4e2db0808731a86cfcf9"
dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48888
I20240502 21:29:14.998739 7213 raft_consensus.cc:595] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:29:15.270573 21750 raft_consensus.cc:979] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714: : Instructing follower 398c3c8f3195492c900439ad5e1931bb to start an election
I20240502 21:29:15.270750 21750 raft_consensus.cc:1067] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Signalling peer 398c3c8f3195492c900439ad5e1931bb to start an election
I20240502 21:29:15.276902 4705 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2fd2328258cf4e2db0808731a86cfcf9"
dest_uuid: "398c3c8f3195492c900439ad5e1931bb"
from {username='slave'} at 127.29.82.69:46901
I20240502 21:29:15.277161 4705 raft_consensus.cc:482] T 2fd2328258cf4e2db0808731a86cfcf9 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:15.277281 4705 raft_consensus.cc:3037] T 2fd2328258cf4e2db0808731a86cfcf9 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:15.286828 4705 raft_consensus.cc:504] T 2fd2328258cf4e2db0808731a86cfcf9 P 398c3c8f3195492c900439ad5e1931bb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:15.294369 7213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd2328258cf4e2db0808731a86cfcf9" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:29:15.294414 4441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd2328258cf4e2db0808731a86cfcf9" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "71e1375b799b439fa3c83ca763c039c1"
I20240502 21:29:15.294622 7213 raft_consensus.cc:3032] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:15.294651 4441 raft_consensus.cc:3037] T 2fd2328258cf4e2db0808731a86cfcf9 P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:15.294703 7213 raft_consensus.cc:729] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:29:15.294953 7213 consensus_queue.cc:260] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:15.295436 7213 raft_consensus.cc:3037] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:15.301069 4441 raft_consensus.cc:2454] T 2fd2328258cf4e2db0808731a86cfcf9 P 71e1375b799b439fa3c83ca763c039c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 2.
I20240502 21:29:15.303051 4705 leader_election.cc:288] T 2fd2328258cf4e2db0808731a86cfcf9 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 2 election: Requested vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:29:15.303236 7213 raft_consensus.cc:2454] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 2.
I20240502 21:29:15.303738 4641 leader_election.cc:302] T 2fd2328258cf4e2db0808731a86cfcf9 P 398c3c8f3195492c900439ad5e1931bb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 398c3c8f3195492c900439ad5e1931bb, 39981a8dfe94462b9a73ba489af4c714, 71e1375b799b439fa3c83ca763c039c1; no voters:
I20240502 21:29:15.303810 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd2328258cf4e2db0808731a86cfcf9" candidate_uuid: "398c3c8f3195492c900439ad5e1931bb" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:29:15.304080 22933 raft_consensus.cc:2781] T 2fd2328258cf4e2db0808731a86cfcf9 P 398c3c8f3195492c900439ad5e1931bb [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:15.304075 7618 raft_consensus.cc:3037] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:15.304234 22933 raft_consensus.cc:686] T 2fd2328258cf4e2db0808731a86cfcf9 P 398c3c8f3195492c900439ad5e1931bb [term 2 LEADER]: Becoming Leader. State: Replica: 398c3c8f3195492c900439ad5e1931bb, State: Running, Role: LEADER
I20240502 21:29:15.304531 22933 consensus_queue.cc:237] T 2fd2328258cf4e2db0808731a86cfcf9 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:15.309872 7618 raft_consensus.cc:2454] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398c3c8f3195492c900439ad5e1931bb in term 2.
I20240502 21:29:15.321638 4291 catalog_manager.cc:5484] T 2fd2328258cf4e2db0808731a86cfcf9 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: term changed from 1 to 2, leader changed from 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) to 398c3c8f3195492c900439ad5e1931bb (127.29.82.67). New cstate: current_term: 2 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:29:15.359015 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:15.396365 4686 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 21:29:15.422865 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:15.430822 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:15.454535 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:29:15.525112 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:15.568257 7194 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 21:29:15.586241 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:15.684540 7618 raft_consensus.cc:1261] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 21:29:15.685611 22933 consensus_queue.cc:1035] T 2fd2328258cf4e2db0808731a86cfcf9 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:15.839411 7214 raft_consensus.cc:1261] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [term 2 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 21:29:15.840194 22933 consensus_queue.cc:1035] T 2fd2328258cf4e2db0808731a86cfcf9 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:15.844177 4441 raft_consensus.cc:1261] T 2fd2328258cf4e2db0808731a86cfcf9 P 71e1375b799b439fa3c83ca763c039c1 [term 2 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 21:29:15.845288 22575 consensus_queue.cc:1035] T 2fd2328258cf4e2db0808731a86cfcf9 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 21:29:16.158205 7285 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.66:35781, user_credentials={real_user=slave}} blocked reactor thread for 47328.8us
I20240502 21:29:16.260474 4686 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 21:29:16.271327 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:16.279903 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:16.323397 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:16.332298 7194 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 21:29:16.337321 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:16.361146 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:16.457417 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:29:16.580022 7618 consensus_queue.cc:237] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:29:16.581892 4439 raft_consensus.cc:1261] T 25be332e341d4869b2165da105c17a8a P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:29:16.582458 4705 raft_consensus.cc:1261] T 25be332e341d4869b2165da105c17a8a P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:29:16.582496 22487 consensus_queue.cc:1035] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:16.583019 22941 consensus_queue.cc:1035] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:16.585464 22941 raft_consensus.cc:2932] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:29:16.588817 4439 raft_consensus.cc:2932] T 25be332e341d4869b2165da105c17a8a P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:29:16.589524 4703 raft_consensus.cc:2932] T 25be332e341d4869b2165da105c17a8a P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:29:16.595721 4294 catalog_manager.cc:5484] T 25be332e341d4869b2165da105c17a8a P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:29:16.605182 7328 tablet_service.cc:1514] Processing DeleteTablet for tablet 25be332e341d4869b2165da105c17a8a with delete_type TABLET_DATA_TOMBSTONED (TS d0456c5e55a542128688904e003d403c not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47078
I20240502 21:29:16.605790 23078 tablet_replica.cc:330] T 25be332e341d4869b2165da105c17a8a P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:29:16.606000 23078 raft_consensus.cc:2229] T 25be332e341d4869b2165da105c17a8a P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:16.606292 23078 raft_consensus.cc:2258] T 25be332e341d4869b2165da105c17a8a P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:16.606922 23078 ts_tablet_manager.cc:1875] T 25be332e341d4869b2165da105c17a8a P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:16.613803 23078 ts_tablet_manager.cc:1888] T 25be332e341d4869b2165da105c17a8a P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:29:16.613996 23078 log.cc:1196] T 25be332e341d4869b2165da105c17a8a P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/25be332e341d4869b2165da105c17a8a
I20240502 21:29:16.615700 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet 25be332e341d4869b2165da105c17a8a (table rebalancer_extra_table_64 [id=8bc8342ce686477cac74df1ebaa3656a]) successfully deleted
I20240502 21:29:17.009863 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:17.027420 4686 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 21:29:17.106907 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240502 21:29:17.113380 4772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.69:33193, user_credentials={real_user=slave}} blocked reactor thread for 55037.8us
I20240502 21:29:17.156150 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:17.170351 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:17.173303 7194 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 21:29:17.216989 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:29:17.253404 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:17.420373 4573 consensus_queue.cc:237] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:17.423115 4839 raft_consensus.cc:1261] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:17.423149 7213 raft_consensus.cc:1261] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:17.423691 23163 consensus_queue.cc:1035] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:17.423962 23051 consensus_queue.cc:1035] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:17.427124 22901 raft_consensus.cc:2932] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:17.427822 7213 raft_consensus.cc:2932] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:17.436131 4291 catalog_manager.cc:5484] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:17.436447 4837 raft_consensus.cc:2932] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:17.450433 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dc27c0c9cee447f84e4bc1a63f64b68 with delete_type TABLET_DATA_TOMBSTONED (TS 398c3c8f3195492c900439ad5e1931bb not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44764
I20240502 21:29:17.451651 23174 tablet_replica.cc:330] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:29:17.451889 23174 raft_consensus.cc:2229] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:17.452217 23174 raft_consensus.cc:2258] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:17.453090 23174 ts_tablet_manager.cc:1875] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:17.468815 23174 ts_tablet_manager.cc:1888] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:29:17.469030 23174 log.cc:1196] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/4dc27c0c9cee447f84e4bc1a63f64b68
I20240502 21:29:17.470070 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet 4dc27c0c9cee447f84e4bc1a63f64b68 (table rebalance_test_table_17 [id=cc837d2089e24e0fa0a7c398ca3eeca8]) successfully deleted
I20240502 21:29:17.570806 7483 consensus_queue.cc:237] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:17.574062 7618 raft_consensus.cc:1261] T a85e7872c28a410496f31ecff9c71edb P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:29:17.574229 7419 consensus_peers.cc:485] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 -> Peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Couldn't send request to peer 39981a8dfe94462b9a73ba489af4c714. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a85e7872c28a410496f31ecff9c71edb. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:17.574896 4439 raft_consensus.cc:1261] T a85e7872c28a410496f31ecff9c71edb P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:29:17.575253 23016 consensus_queue.cc:1035] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:29:17.575603 23211 consensus_queue.cc:1035] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:29:17.581912 23212 raft_consensus.cc:2932] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:17.582674 4439 raft_consensus.cc:2932] T a85e7872c28a410496f31ecff9c71edb P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:17.587276 7618 raft_consensus.cc:2932] T a85e7872c28a410496f31ecff9c71edb P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:17.590042 4296 catalog_manager.cc:5484] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 reported cstate change: config changed from index -1 to 2, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:17.634783 4439 consensus_queue.cc:237] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:29:17.637850 4573 raft_consensus.cc:1261] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:17.637866 4705 raft_consensus.cc:1261] T 14bd28b4c6a54d98a18351b60f299a3a P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:17.638424 23233 consensus_queue.cc:1035] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:29:17.638814 4377 consensus_peers.cc:485] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 -> Peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Couldn't send request to peer d0456c5e55a542128688904e003d403c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14bd28b4c6a54d98a18351b60f299a3a. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:17.638953 23012 consensus_queue.cc:1035] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:17.643247 23233 raft_consensus.cc:2932] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:29:17.643817 4705 raft_consensus.cc:2932] T 14bd28b4c6a54d98a18351b60f299a3a P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:29:17.651365 4292 catalog_manager.cc:5484] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: config changed from index -1 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:29:17.654254 4575 raft_consensus.cc:2932] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:29:18.040009 23288 ts_tablet_manager.cc:902] T a85e7872c28a410496f31ecff9c71edb P 39981a8dfe94462b9a73ba489af4c714: Initiating tablet copy from peer 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:29:18.116501 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:18.120132 23288 tablet_copy_client.cc:308] T a85e7872c28a410496f31ecff9c71edb P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.71:33913
I20240502 21:29:18.120900 7505 tablet_copy_service.cc:140] P 31a72286397c4edc9a63eca3412be168: Received BeginTabletCopySession request for tablet a85e7872c28a410496f31ecff9c71edb from peer 39981a8dfe94462b9a73ba489af4c714 ({username='slave'} at 127.29.82.69:38761)
I20240502 21:29:18.121043 7505 tablet_copy_service.cc:161] P 31a72286397c4edc9a63eca3412be168: Beginning new tablet copy session on tablet a85e7872c28a410496f31ecff9c71edb from peer 39981a8dfe94462b9a73ba489af4c714 at {username='slave'} at 127.29.82.69:38761: session id = 39981a8dfe94462b9a73ba489af4c714-a85e7872c28a410496f31ecff9c71edb
I20240502 21:29:18.124065 7505 tablet_copy_source_session.cc:205] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:18.149231 4686 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 21:29:18.161980 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:18.163681 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:18.164551 7194 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 21:29:18.174129 23288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a85e7872c28a410496f31ecff9c71edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:18.189641 23288 tablet_copy_client.cc:783] T a85e7872c28a410496f31ecff9c71edb P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:18.189883 23288 tablet_copy_client.cc:647] T a85e7872c28a410496f31ecff9c71edb P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:18.199205 23288 tablet_copy_client.cc:523] T a85e7872c28a410496f31ecff9c71edb P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:18.203874 23288 tablet_bootstrap.cc:492] T a85e7872c28a410496f31ecff9c71edb P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:29:18.234269 23309 ts_tablet_manager.cc:902] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c: Initiating tablet copy from peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:29:18.245815 23288 tablet_bootstrap.cc:492] T a85e7872c28a410496f31ecff9c71edb P 39981a8dfe94462b9a73ba489af4c714: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:18.246192 23288 tablet_bootstrap.cc:492] T a85e7872c28a410496f31ecff9c71edb P 39981a8dfe94462b9a73ba489af4c714: Bootstrap complete.
I20240502 21:29:18.246439 23288 ts_tablet_manager.cc:1367] T a85e7872c28a410496f31ecff9c71edb P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.043s user 0.009s sys 0.005s
I20240502 21:29:18.247196 23288 raft_consensus.cc:348] T a85e7872c28a410496f31ecff9c71edb P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:18.247704 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:18.247408 23288 raft_consensus.cc:729] T a85e7872c28a410496f31ecff9c71edb P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:29:18.249965 23288 consensus_queue.cc:260] T a85e7872c28a410496f31ecff9c71edb P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:18.252538 23288 ts_tablet_manager.cc:1398] T a85e7872c28a410496f31ecff9c71edb P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 21:29:18.253296 7505 tablet_copy_service.cc:342] P 31a72286397c4edc9a63eca3412be168: Request end of tablet copy session 39981a8dfe94462b9a73ba489af4c714-a85e7872c28a410496f31ecff9c71edb received from {username='slave'} at 127.29.82.69:38761
I20240502 21:29:18.253396 7505 tablet_copy_service.cc:433] P 31a72286397c4edc9a63eca3412be168: ending tablet copy session 39981a8dfe94462b9a73ba489af4c714-a85e7872c28a410496f31ecff9c71edb on tablet a85e7872c28a410496f31ecff9c71edb with peer 39981a8dfe94462b9a73ba489af4c714
I20240502 21:29:18.298664 23309 tablet_copy_client.cc:308] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.65:42781
I20240502 21:29:18.305424 4463 tablet_copy_service.cc:140] P 71e1375b799b439fa3c83ca763c039c1: Received BeginTabletCopySession request for tablet 14bd28b4c6a54d98a18351b60f299a3a from peer d0456c5e55a542128688904e003d403c ({username='slave'} at 127.29.82.70:41689)
I20240502 21:29:18.305630 4463 tablet_copy_service.cc:161] P 71e1375b799b439fa3c83ca763c039c1: Beginning new tablet copy session on tablet 14bd28b4c6a54d98a18351b60f299a3a from peer d0456c5e55a542128688904e003d403c at {username='slave'} at 127.29.82.70:41689: session id = d0456c5e55a542128688904e003d403c-14bd28b4c6a54d98a18351b60f299a3a
I20240502 21:29:18.309162 4463 tablet_copy_source_session.cc:205] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:29:18.310742 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:29:18.324991 23309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14bd28b4c6a54d98a18351b60f299a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:18.333257 23309 tablet_copy_client.cc:783] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c: tablet copy: Starting download of 5 data blocks...
I20240502 21:29:18.350869 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:18.379678 23309 tablet_copy_client.cc:647] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:18.382201 23309 tablet_copy_client.cc:523] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:18.386135 23309 tablet_bootstrap.cc:492] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:29:18.426716 23309 tablet_bootstrap.cc:492] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:18.427639 23309 tablet_bootstrap.cc:492] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c: Bootstrap complete.
I20240502 21:29:18.428583 23309 ts_tablet_manager.cc:1367] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.043s user 0.017s sys 0.001s
I20240502 21:29:18.429615 23309 raft_consensus.cc:348] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:29:18.429904 23309 raft_consensus.cc:729] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:29:18.430258 23309 consensus_queue.cc:260] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:29:18.430986 23309 ts_tablet_manager.cc:1398] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:29:18.434821 4463 tablet_copy_service.cc:342] P 71e1375b799b439fa3c83ca763c039c1: Request end of tablet copy session d0456c5e55a542128688904e003d403c-14bd28b4c6a54d98a18351b60f299a3a received from {username='slave'} at 127.29.82.70:41689
I20240502 21:29:18.434952 4463 tablet_copy_service.cc:433] P 71e1375b799b439fa3c83ca763c039c1: ending tablet copy session d0456c5e55a542128688904e003d403c-14bd28b4c6a54d98a18351b60f299a3a on tablet 14bd28b4c6a54d98a18351b60f299a3a with peer d0456c5e55a542128688904e003d403c
I20240502 21:29:18.568810 7213 raft_consensus.cc:1203] T a85e7872c28a410496f31ecff9c71edb P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:29:18.638506 7396 maintenance_manager.cc:419] P d0456c5e55a542128688904e003d403c: Scheduling UndoDeltaBlockGCOp(14bd28b4c6a54d98a18351b60f299a3a): 4925 bytes on disk
I20240502 21:29:18.639245 7286 maintenance_manager.cc:643] P d0456c5e55a542128688904e003d403c: UndoDeltaBlockGCOp(14bd28b4c6a54d98a18351b60f299a3a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
I20240502 21:29:18.735026 7347 raft_consensus.cc:1203] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:29:18.754941 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:18.757519 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:18.812855 7194 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 21:29:18.879001 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 21:29:18.888000 7420 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.66:35781, user_credentials={real_user=slave}} blocked reactor thread for 51801.3us
W20240502 21:29:18.899674 4509 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.72:41223, user_credentials={real_user=slave}} blocked reactor thread for 45456.5us
I20240502 21:29:18.933393 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:18.935905 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240502 21:29:18.953569 4641 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.70:40719, user_credentials={real_user=slave}} blocked reactor thread for 182497us
I20240502 21:29:18.988991 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:29:19.017690 4686 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 21:29:19.036355 7549 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.65:42781, user_credentials={real_user=slave}} blocked reactor thread for 51618.2us
W20240502 21:29:19.037439 7552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.67:34537, user_credentials={real_user=slave}} blocked reactor thread for 54568.9us
I20240502 21:29:19.281246 7618 consensus_queue.cc:237] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:29:19.283836 4704 raft_consensus.cc:1261] T 6d0334be884c45c3bd9e49095454dfb1 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:29:19.284179 7484 raft_consensus.cc:1261] T 6d0334be884c45c3bd9e49095454dfb1 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:29:19.284432 23394 consensus_queue.cc:1035] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:19.284698 23391 consensus_queue.cc:1035] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:19.287393 23391 raft_consensus.cc:2932] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:29:19.288255 7484 raft_consensus.cc:2932] T 6d0334be884c45c3bd9e49095454dfb1 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:29:19.288424 4704 raft_consensus.cc:2932] T 6d0334be884c45c3bd9e49095454dfb1 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:29:19.296454 4294 catalog_manager.cc:5484] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: config changed from index 2 to 3, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) evicted. New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:29:19.318850 7328 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d0334be884c45c3bd9e49095454dfb1 with delete_type TABLET_DATA_TOMBSTONED (TS d0456c5e55a542128688904e003d403c not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47078
I20240502 21:29:19.321777 23427 tablet_replica.cc:330] T 6d0334be884c45c3bd9e49095454dfb1 P d0456c5e55a542128688904e003d403c: stopping tablet replica
I20240502 21:29:19.322154 23427 raft_consensus.cc:2229] T 6d0334be884c45c3bd9e49095454dfb1 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:19.322458 23427 raft_consensus.cc:2258] T 6d0334be884c45c3bd9e49095454dfb1 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:19.323223 23427 ts_tablet_manager.cc:1875] T 6d0334be884c45c3bd9e49095454dfb1 P d0456c5e55a542128688904e003d403c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:19.330413 23427 ts_tablet_manager.cc:1888] T 6d0334be884c45c3bd9e49095454dfb1 P d0456c5e55a542128688904e003d403c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:29:19.330597 23427 log.cc:1196] T 6d0334be884c45c3bd9e49095454dfb1 P d0456c5e55a542128688904e003d403c: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/wals/6d0334be884c45c3bd9e49095454dfb1
I20240502 21:29:19.331513 4286 catalog_manager.cc:4830] TS d0456c5e55a542128688904e003d403c (127.29.82.70:40719): tablet 6d0334be884c45c3bd9e49095454dfb1 (table rebalancer_extra_table_4 [id=2ba724ea31944444b73e3c52e98f06e7]) successfully deleted
I20240502 21:29:19.688732 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:19.765498 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:19.788862 4686 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 21:29:19.817641 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:19.818120 7194 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 21:29:19.818985 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:19.876892 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:29:19.908878 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:20.085213 4704 consensus_queue.cc:237] T 2fd2328258cf4e2db0808731a86cfcf9 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:20.088766 7618 raft_consensus.cc:1261] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:29:20.089136 4439 raft_consensus.cc:1261] T 2fd2328258cf4e2db0808731a86cfcf9 P 71e1375b799b439fa3c83ca763c039c1 [term 2 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:29:20.089995 22933 consensus_queue.cc:1035] T 2fd2328258cf4e2db0808731a86cfcf9 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:20.093235 22933 consensus_queue.cc:1035] T 2fd2328258cf4e2db0808731a86cfcf9 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:20.093858 22933 raft_consensus.cc:2932] T 2fd2328258cf4e2db0808731a86cfcf9 P 398c3c8f3195492c900439ad5e1931bb [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:29:20.100651 7618 raft_consensus.cc:2932] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:29:20.104897 4439 raft_consensus.cc:2932] T 2fd2328258cf4e2db0808731a86cfcf9 P 71e1375b799b439fa3c83ca763c039c1 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:29:20.109117 4292 catalog_manager.cc:5484] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: config changed from index 2 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) evicted. New cstate: current_term: 2 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:29:20.117846 7194 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fd2328258cf4e2db0808731a86cfcf9 with delete_type TABLET_DATA_TOMBSTONED (TS 39981a8dfe94462b9a73ba489af4c714 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44652
I20240502 21:29:20.118531 23537 tablet_replica.cc:330] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714: stopping tablet replica
I20240502 21:29:20.118736 23537 raft_consensus.cc:2229] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:20.118948 23537 raft_consensus.cc:2258] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:20.119499 23537 ts_tablet_manager.cc:1875] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:20.124924 23537 ts_tablet_manager.cc:1888] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240502 21:29:20.125075 23537 log.cc:1196] T 2fd2328258cf4e2db0808731a86cfcf9 P 39981a8dfe94462b9a73ba489af4c714: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/wals/2fd2328258cf4e2db0808731a86cfcf9
I20240502 21:29:20.125898 4286 catalog_manager.cc:4830] TS 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): tablet 2fd2328258cf4e2db0808731a86cfcf9 (table rebalancer_extra_table_54 [id=ec5b364156ae4e6ebd1fdaca42b585f6]) successfully deleted
I20240502 21:29:20.422874 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:20.483193 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:20.487463 4686 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 21:29:20.488451 7194 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 21:29:20.538640 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:20.587741 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:20.686722 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:20.697146 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:29:20.813447 4439 consensus_queue.cc:237] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:29:20.815948 4575 raft_consensus.cc:1261] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:20.816543 23011 consensus_queue.cc:1035] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:20.816702 7349 raft_consensus.cc:1261] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:20.817255 23011 consensus_queue.cc:1035] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:20.820305 23012 raft_consensus.cc:2932] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:29:20.821135 4575 raft_consensus.cc:2932] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:29:20.827809 4297 catalog_manager.cc:5484] T 14bd28b4c6a54d98a18351b60f299a3a P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:29:20.868549 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet 14bd28b4c6a54d98a18351b60f299a3a with delete_type TABLET_DATA_TOMBSTONED (TS 398c3c8f3195492c900439ad5e1931bb not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44764
I20240502 21:29:20.869217 23627 tablet_replica.cc:330] T 14bd28b4c6a54d98a18351b60f299a3a P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:29:20.869453 23627 raft_consensus.cc:2229] T 14bd28b4c6a54d98a18351b60f299a3a P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:20.869727 23627 raft_consensus.cc:2258] T 14bd28b4c6a54d98a18351b60f299a3a P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:20.870570 23627 ts_tablet_manager.cc:1875] T 14bd28b4c6a54d98a18351b60f299a3a P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 21:29:20.874971 7282 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.72:41223, user_credentials={real_user=slave}} blocked reactor thread for 45087.4us
I20240502 21:29:20.877171 7349 raft_consensus.cc:2932] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:29:20.906327 23627 ts_tablet_manager.cc:1888] T 14bd28b4c6a54d98a18351b60f299a3a P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:29:20.906463 23627 log.cc:1196] T 14bd28b4c6a54d98a18351b60f299a3a P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/14bd28b4c6a54d98a18351b60f299a3a
I20240502 21:29:20.907275 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet 14bd28b4c6a54d98a18351b60f299a3a (table rebalance_test_table_3 [id=3c51dfaf2d824f93813d8d39758c9a8c]) successfully deleted
I20240502 21:29:20.963601 4439 consensus_queue.cc:237] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:29:20.965801 7618 raft_consensus.cc:1261] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:29:20.965991 4705 raft_consensus.cc:1261] T 1bddb5e40d464de082c1d586618c86d7 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:29:20.966392 23011 consensus_queue.cc:1035] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:29:20.966764 4377 consensus_peers.cc:485] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1 -> Peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Couldn't send request to peer d0456c5e55a542128688904e003d403c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1bddb5e40d464de082c1d586618c86d7. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:20.967007 23012 consensus_queue.cc:1035] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:29:20.970827 23012 raft_consensus.cc:2932] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:29:20.976725 4705 raft_consensus.cc:2932] T 1bddb5e40d464de082c1d586618c86d7 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:29:20.986963 7618 raft_consensus.cc:2932] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:29:20.988461 4297 catalog_manager.cc:5484] T 1bddb5e40d464de082c1d586618c86d7 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: config changed from index -1 to 2, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:29:21.025781 4575 consensus_queue.cc:237] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:21.028184 4705 raft_consensus.cc:1261] T 81c2664e2d45448b9e08a6fac1c90a0d P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:21.028307 4837 raft_consensus.cc:1261] T 81c2664e2d45448b9e08a6fac1c90a0d P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:29:21.028692 4510 consensus_peers.cc:485] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8 -> Peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Couldn't send request to peer 39981a8dfe94462b9a73ba489af4c714. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81c2664e2d45448b9e08a6fac1c90a0d. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:21.028803 22901 consensus_queue.cc:1035] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:21.029281 22901 consensus_queue.cc:1035] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:21.031939 22901 raft_consensus.cc:2932] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:21.032433 4837 raft_consensus.cc:2932] T 81c2664e2d45448b9e08a6fac1c90a0d P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:21.039393 4294 catalog_manager.cc:5484] T 81c2664e2d45448b9e08a6fac1c90a0d P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index -1 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:21.042316 4704 raft_consensus.cc:2932] T 81c2664e2d45448b9e08a6fac1c90a0d P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:21.340936 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:21.356215 4686 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 21:29:21.383816 7194 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 21:29:21.437103 23731 ts_tablet_manager.cc:902] T 1bddb5e40d464de082c1d586618c86d7 P d0456c5e55a542128688904e003d403c: Initiating tablet copy from peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:29:21.440203 23732 ts_tablet_manager.cc:902] T 81c2664e2d45448b9e08a6fac1c90a0d P 39981a8dfe94462b9a73ba489af4c714: Initiating tablet copy from peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:29:21.451020 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:21.468108 23732 tablet_copy_client.cc:308] T 81c2664e2d45448b9e08a6fac1c90a0d P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.66:35781
I20240502 21:29:21.471443 4595 tablet_copy_service.cc:140] P 7a736a8ebb164ce0a051c0fccd76c3b8: Received BeginTabletCopySession request for tablet 81c2664e2d45448b9e08a6fac1c90a0d from peer 39981a8dfe94462b9a73ba489af4c714 ({username='slave'} at 127.29.82.69:37791)
I20240502 21:29:21.471637 4595 tablet_copy_service.cc:161] P 7a736a8ebb164ce0a051c0fccd76c3b8: Beginning new tablet copy session on tablet 81c2664e2d45448b9e08a6fac1c90a0d from peer 39981a8dfe94462b9a73ba489af4c714 at {username='slave'} at 127.29.82.69:37791: session id = 39981a8dfe94462b9a73ba489af4c714-81c2664e2d45448b9e08a6fac1c90a0d
I20240502 21:29:21.474965 4595 tablet_copy_source_session.cc:205] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:29:21.476439 23732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81c2664e2d45448b9e08a6fac1c90a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:21.476791 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:21.482926 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:21.488899 23732 tablet_copy_client.cc:783] T 81c2664e2d45448b9e08a6fac1c90a0d P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Starting download of 5 data blocks...
I20240502 21:29:21.500003 23731 tablet_copy_client.cc:308] T 1bddb5e40d464de082c1d586618c86d7 P d0456c5e55a542128688904e003d403c: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.65:42781
I20240502 21:29:21.500872 4463 tablet_copy_service.cc:140] P 71e1375b799b439fa3c83ca763c039c1: Received BeginTabletCopySession request for tablet 1bddb5e40d464de082c1d586618c86d7 from peer d0456c5e55a542128688904e003d403c ({username='slave'} at 127.29.82.70:41689)
I20240502 21:29:21.501031 4463 tablet_copy_service.cc:161] P 71e1375b799b439fa3c83ca763c039c1: Beginning new tablet copy session on tablet 1bddb5e40d464de082c1d586618c86d7 from peer d0456c5e55a542128688904e003d403c at {username='slave'} at 127.29.82.70:41689: session id = d0456c5e55a542128688904e003d403c-1bddb5e40d464de082c1d586618c86d7
I20240502 21:29:21.503943 4463 tablet_copy_source_session.cc:205] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:21.505270 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:21.510936 23731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bddb5e40d464de082c1d586618c86d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:21.514358 23732 tablet_copy_client.cc:647] T 81c2664e2d45448b9e08a6fac1c90a0d P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:21.518728 23732 tablet_copy_client.cc:523] T 81c2664e2d45448b9e08a6fac1c90a0d P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:21.522337 23732 tablet_bootstrap.cc:492] T 81c2664e2d45448b9e08a6fac1c90a0d P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:29:21.551323 23732 tablet_bootstrap.cc:492] T 81c2664e2d45448b9e08a6fac1c90a0d P 39981a8dfe94462b9a73ba489af4c714: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:21.554044 23731 tablet_copy_client.cc:783] T 1bddb5e40d464de082c1d586618c86d7 P d0456c5e55a542128688904e003d403c: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:21.554250 23731 tablet_copy_client.cc:647] T 1bddb5e40d464de082c1d586618c86d7 P d0456c5e55a542128688904e003d403c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:21.561996 23732 tablet_bootstrap.cc:492] T 81c2664e2d45448b9e08a6fac1c90a0d P 39981a8dfe94462b9a73ba489af4c714: Bootstrap complete.
I20240502 21:29:21.562255 23732 ts_tablet_manager.cc:1367] T 81c2664e2d45448b9e08a6fac1c90a0d P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.040s user 0.016s sys 0.004s
I20240502 21:29:21.563656 23732 raft_consensus.cc:348] T 81c2664e2d45448b9e08a6fac1c90a0d P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:21.563885 23732 raft_consensus.cc:729] T 81c2664e2d45448b9e08a6fac1c90a0d P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:29:21.566241 23732 consensus_queue.cc:260] T 81c2664e2d45448b9e08a6fac1c90a0d P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:21.569972 23732 ts_tablet_manager.cc:1398] T 81c2664e2d45448b9e08a6fac1c90a0d P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240502 21:29:21.570957 4595 tablet_copy_service.cc:342] P 7a736a8ebb164ce0a051c0fccd76c3b8: Request end of tablet copy session 39981a8dfe94462b9a73ba489af4c714-81c2664e2d45448b9e08a6fac1c90a0d received from {username='slave'} at 127.29.82.69:37791
I20240502 21:29:21.571100 4595 tablet_copy_service.cc:433] P 7a736a8ebb164ce0a051c0fccd76c3b8: ending tablet copy session 39981a8dfe94462b9a73ba489af4c714-81c2664e2d45448b9e08a6fac1c90a0d on tablet 81c2664e2d45448b9e08a6fac1c90a0d with peer 39981a8dfe94462b9a73ba489af4c714
I20240502 21:29:21.577733 23731 tablet_copy_client.cc:523] T 1bddb5e40d464de082c1d586618c86d7 P d0456c5e55a542128688904e003d403c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 21:29:21.586795 4639 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.65:42781, user_credentials={real_user=slave}} blocked reactor thread for 53258.1us
I20240502 21:29:21.609113 23731 tablet_bootstrap.cc:492] T 1bddb5e40d464de082c1d586618c86d7 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
I20240502 21:29:21.619892 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:29:21.637053 23731 tablet_bootstrap.cc:492] T 1bddb5e40d464de082c1d586618c86d7 P d0456c5e55a542128688904e003d403c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:21.637530 23731 tablet_bootstrap.cc:492] T 1bddb5e40d464de082c1d586618c86d7 P d0456c5e55a542128688904e003d403c: Bootstrap complete.
I20240502 21:29:21.637790 23731 ts_tablet_manager.cc:1367] T 1bddb5e40d464de082c1d586618c86d7 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.006s
I20240502 21:29:21.638536 23731 raft_consensus.cc:348] T 1bddb5e40d464de082c1d586618c86d7 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:29:21.638767 23731 raft_consensus.cc:729] T 1bddb5e40d464de082c1d586618c86d7 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
I20240502 21:29:21.639043 23731 consensus_queue.cc:260] T 1bddb5e40d464de082c1d586618c86d7 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:29:21.639693 23731 ts_tablet_manager.cc:1398] T 1bddb5e40d464de082c1d586618c86d7 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:29:21.641053 4463 tablet_copy_service.cc:342] P 71e1375b799b439fa3c83ca763c039c1: Request end of tablet copy session d0456c5e55a542128688904e003d403c-1bddb5e40d464de082c1d586618c86d7 received from {username='slave'} at 127.29.82.70:41689
I20240502 21:29:21.641185 4463 tablet_copy_service.cc:433] P 71e1375b799b439fa3c83ca763c039c1: ending tablet copy session d0456c5e55a542128688904e003d403c-1bddb5e40d464de082c1d586618c86d7 on tablet 1bddb5e40d464de082c1d586618c86d7 with peer d0456c5e55a542128688904e003d403c
I20240502 21:29:21.700874 7262 maintenance_manager.cc:419] P 39981a8dfe94462b9a73ba489af4c714: Scheduling UndoDeltaBlockGCOp(81c2664e2d45448b9e08a6fac1c90a0d): 4586 bytes on disk
I20240502 21:29:21.701567 7152 maintenance_manager.cc:643] P 39981a8dfe94462b9a73ba489af4c714: UndoDeltaBlockGCOp(81c2664e2d45448b9e08a6fac1c90a0d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240502 21:29:21.963012 7349 raft_consensus.cc:1203] T 1bddb5e40d464de082c1d586618c86d7 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:29:21.990978 7213 raft_consensus.cc:1203] T 81c2664e2d45448b9e08a6fac1c90a0d P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:29:22.142226 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:22.210727 7194 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 21:29:22.213189 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:22.214291 4686 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 21:29:22.253680 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:22.288890 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:22.308770 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:22.321053 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:29:22.532802 7484 consensus_queue.cc:237] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:22.535262 4441 raft_consensus.cc:1261] T a85e7872c28a410496f31ecff9c71edb P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:29:22.535815 23385 consensus_queue.cc:1035] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:22.536187 7213 raft_consensus.cc:1261] T a85e7872c28a410496f31ecff9c71edb P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:29:22.536690 23385 consensus_queue.cc:1035] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:22.539212 23491 raft_consensus.cc:2932] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:22.542683 4441 raft_consensus.cc:2932] T a85e7872c28a410496f31ecff9c71edb P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:22.544730 7213 raft_consensus.cc:2932] T a85e7872c28a410496f31ecff9c71edb P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:22.549351 4291 catalog_manager.cc:5484] T a85e7872c28a410496f31ecff9c71edb P 71e1375b799b439fa3c83ca763c039c1 reported cstate change: config changed from index 2 to 3, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) evicted. New cstate: current_term: 1 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:22.559865 7597 tablet_service.cc:1514] Processing DeleteTablet for tablet a85e7872c28a410496f31ecff9c71edb with delete_type TABLET_DATA_TOMBSTONED (TS 0ddd486109f5434cace13d60d633ac04 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44444
I20240502 21:29:22.567997 23854 tablet_replica.cc:330] T a85e7872c28a410496f31ecff9c71edb P 0ddd486109f5434cace13d60d633ac04: stopping tablet replica
I20240502 21:29:22.568257 23854 raft_consensus.cc:2229] T a85e7872c28a410496f31ecff9c71edb P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:22.568516 23854 raft_consensus.cc:2258] T a85e7872c28a410496f31ecff9c71edb P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:22.569298 23854 ts_tablet_manager.cc:1875] T a85e7872c28a410496f31ecff9c71edb P 0ddd486109f5434cace13d60d633ac04: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:22.577982 23854 ts_tablet_manager.cc:1888] T a85e7872c28a410496f31ecff9c71edb P 0ddd486109f5434cace13d60d633ac04: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:29:22.578107 23854 log.cc:1196] T a85e7872c28a410496f31ecff9c71edb P 0ddd486109f5434cace13d60d633ac04: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/wals/a85e7872c28a410496f31ecff9c71edb
I20240502 21:29:22.578853 4285 catalog_manager.cc:4830] TS 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): tablet a85e7872c28a410496f31ecff9c71edb (table rebalancer_extra_table_26 [id=1e6edd8bfe59407fa7863cd780bfae31]) successfully deleted
I20240502 21:29:22.899258 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:22.923729 4686 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 21:29:22.926294 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:22.981856 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:22.985577 7194 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 21:29:23.025065 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:23.058609 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:23.077021 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:29:23.242739 4575 consensus_queue.cc:237] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:23.244757 7213 raft_consensus.cc:1261] T 81c2664e2d45448b9e08a6fac1c90a0d P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:23.244774 4837 raft_consensus.cc:1261] T 81c2664e2d45448b9e08a6fac1c90a0d P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:23.245268 23596 consensus_queue.cc:1035] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:23.245570 23782 consensus_queue.cc:1035] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:23.248389 23596 raft_consensus.cc:2932] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:23.249095 7213 raft_consensus.cc:2932] T 81c2664e2d45448b9e08a6fac1c90a0d P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:23.249424 4837 raft_consensus.cc:2932] T 81c2664e2d45448b9e08a6fac1c90a0d P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:23.256644 4296 catalog_manager.cc:5484] T 81c2664e2d45448b9e08a6fac1c90a0d P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: config changed from index 4 to 5, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New cstate: current_term: 1 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:23.268867 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet 81c2664e2d45448b9e08a6fac1c90a0d with delete_type TABLET_DATA_TOMBSTONED (TS 398c3c8f3195492c900439ad5e1931bb not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44764
I20240502 21:29:23.272962 23944 tablet_replica.cc:330] T 81c2664e2d45448b9e08a6fac1c90a0d P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:29:23.279834 23944 raft_consensus.cc:2229] T 81c2664e2d45448b9e08a6fac1c90a0d P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:23.280191 23944 raft_consensus.cc:2258] T 81c2664e2d45448b9e08a6fac1c90a0d P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:23.281183 23944 ts_tablet_manager.cc:1875] T 81c2664e2d45448b9e08a6fac1c90a0d P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:23.295600 23944 ts_tablet_manager.cc:1888] T 81c2664e2d45448b9e08a6fac1c90a0d P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:29:23.295766 23944 log.cc:1196] T 81c2664e2d45448b9e08a6fac1c90a0d P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/81c2664e2d45448b9e08a6fac1c90a0d
I20240502 21:29:23.296578 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet 81c2664e2d45448b9e08a6fac1c90a0d (table rebalance_test_table_13 [id=d14fe34dcad44629af56956cc9e71e5b]) successfully deleted
I20240502 21:29:23.711620 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:23.739662 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:23.757813 7194 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 21:29:23.758244 4686 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 21:29:23.767540 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:23.825623 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:23.866730 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:29:23.971078 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:24.122568 4441 consensus_queue.cc:237] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:29:24.124872 7618 raft_consensus.cc:1261] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:29:24.125437 23826 consensus_queue.cc:1035] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:24.125689 7349 raft_consensus.cc:1261] T 1bddb5e40d464de082c1d586618c86d7 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:29:24.126437 23985 consensus_queue.cc:1035] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:29:24.128273 23826 raft_consensus.cc:2932] T 1bddb5e40d464de082c1d586618c86d7 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:29:24.130035 7618 raft_consensus.cc:2932] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:29:24.146009 4292 catalog_manager.cc:5484] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: config changed from index 2 to 3, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:29:24.152758 7347 raft_consensus.cc:2932] T 1bddb5e40d464de082c1d586618c86d7 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:29:24.185917 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bddb5e40d464de082c1d586618c86d7 with delete_type TABLET_DATA_TOMBSTONED (TS 398c3c8f3195492c900439ad5e1931bb not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44764
I20240502 21:29:24.191943 24064 tablet_replica.cc:330] T 1bddb5e40d464de082c1d586618c86d7 P 398c3c8f3195492c900439ad5e1931bb: stopping tablet replica
I20240502 21:29:24.192164 24064 raft_consensus.cc:2229] T 1bddb5e40d464de082c1d586618c86d7 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:24.192373 24064 raft_consensus.cc:2258] T 1bddb5e40d464de082c1d586618c86d7 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:24.193032 24064 ts_tablet_manager.cc:1875] T 1bddb5e40d464de082c1d586618c86d7 P 398c3c8f3195492c900439ad5e1931bb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:24.199141 24064 ts_tablet_manager.cc:1888] T 1bddb5e40d464de082c1d586618c86d7 P 398c3c8f3195492c900439ad5e1931bb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 21:29:24.199287 24064 log.cc:1196] T 1bddb5e40d464de082c1d586618c86d7 P 398c3c8f3195492c900439ad5e1931bb: Deleting WAL directory at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/wals/1bddb5e40d464de082c1d586618c86d7
I20240502 21:29:24.200031 4287 catalog_manager.cc:4830] TS 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): tablet 1bddb5e40d464de082c1d586618c86d7 (table rebalancer_extra_table_42 [id=9c0b534f9084406c81e37e510fcd71d7]) successfully deleted
I20240502 21:29:24.568249 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:24.599829 4686 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 21:29:24.616618 4555 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:24.626572 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:24.632267 4421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:24.643208 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:29:24.662021 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:24.679159 7194 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 21:29:24.992441 7212 consensus_queue.cc:237] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:29:24.995604 7349 raft_consensus.cc:1261] T a410b7da18004b23bdad650f59a30e8c P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:29:24.996188 4703 raft_consensus.cc:1261] T a410b7da18004b23bdad650f59a30e8c P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:29:24.996492 23316 consensus_queue.cc:1035] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:29:24.996778 24005 consensus_queue.cc:1035] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:29:24.997030 7148 consensus_peers.cc:485] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714 -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a410b7da18004b23bdad650f59a30e8c. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:24.999511 24150 raft_consensus.cc:2932] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:29:25.002415 4703 raft_consensus.cc:2932] T a410b7da18004b23bdad650f59a30e8c P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:29:25.005379 7349 raft_consensus.cc:2932] T a410b7da18004b23bdad650f59a30e8c P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:29:25.011009 4300 catalog_manager.cc:5484] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: config changed from index -1 to 2, VOTER 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) added. New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
I20240502 21:29:25.075696 4703 consensus_queue.cc:237] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:25.078758 4573 raft_consensus.cc:1261] T 2285212d78b0460ca329949eb3140e69 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:25.078816 7617 raft_consensus.cc:1261] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 398c3c8f3195492c900439ad5e1931bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:25.079937 23526 consensus_queue.cc:1035] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:25.080653 23526 consensus_queue.cc:1035] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 21:29:25.081506 4639 consensus_peers.cc:485] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb -> Peer 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): Couldn't send request to peer 31a72286397c4edc9a63eca3412be168. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2285212d78b0460ca329949eb3140e69. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:25.095235 23526 raft_consensus.cc:2932] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:25.095938 4573 raft_consensus.cc:2932] T 2285212d78b0460ca329949eb3140e69 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:25.104040 7617 raft_consensus.cc:2932] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:25.105772 4291 catalog_manager.cc:5484] T 2285212d78b0460ca329949eb3140e69 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: config changed from index 5 to 6, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New cstate: current_term: 1 leader_uuid: "398c3c8f3195492c900439ad5e1931bb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:25.181407 4837 consensus_queue.cc:237] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:29:25.184316 4703 raft_consensus.cc:1261] T 686a74b8c1034112b293c4c7c6bcd99a P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:25.184499 7483 raft_consensus.cc:1261] T 686a74b8c1034112b293c4c7c6bcd99a P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:29:25.184651 4770 consensus_peers.cc:485] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 686a74b8c1034112b293c4c7c6bcd99a. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:25.184988 23735 consensus_queue.cc:1035] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:25.185272 24006 consensus_queue.cc:1035] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:25.197867 24006 raft_consensus.cc:2932] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:29:25.198509 7484 raft_consensus.cc:2932] T 686a74b8c1034112b293c4c7c6bcd99a P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:29:25.204459 4703 raft_consensus.cc:2932] T 686a74b8c1034112b293c4c7c6bcd99a P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:29:25.205927 4291 catalog_manager.cc:5484] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index 3 to 4, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:29:25.246249 7349 consensus_queue.cc:237] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:29:25.249186 4573 raft_consensus.cc:1261] T c6459a8c391249529d2672ab6c62a8af P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:29:25.249533 7285 consensus_peers.cc:485] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6459a8c391249529d2672ab6c62a8af. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:25.249663 7617 raft_consensus.cc:1261] T c6459a8c391249529d2672ab6c62a8af P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:25.249836 24031 consensus_queue.cc:1035] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:25.250531 24215 consensus_queue.cc:1035] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:25.253506 24031 raft_consensus.cc:2932] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:29:25.254077 7617 raft_consensus.cc:2932] T c6459a8c391249529d2672ab6c62a8af P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:29:25.263440 4300 catalog_manager.cc:5484] T c6459a8c391249529d2672ab6c62a8af P 0ddd486109f5434cace13d60d633ac04 reported cstate change: config changed from index 3 to 4, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New cstate: current_term: 1 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:29:25.265966 4573 raft_consensus.cc:2932] T c6459a8c391249529d2672ab6c62a8af P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
I20240502 21:29:25.303936 4837 consensus_queue.cc:237] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:25.306380 7349 raft_consensus.cc:1261] T dd1cf8e447e54d6e907ffa0e4681515f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:25.306367 4573 raft_consensus.cc:1261] T dd1cf8e447e54d6e907ffa0e4681515f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 21:29:25.306516 4772 consensus_peers.cc:485] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Couldn't send request to peer 39981a8dfe94462b9a73ba489af4c714. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd1cf8e447e54d6e907ffa0e4681515f. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:25.306926 24027 consensus_queue.cc:1035] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:25.307224 23735 consensus_queue.cc:1035] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:25.310019 24027 raft_consensus.cc:2932] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:25.310703 4573 raft_consensus.cc:2932] T dd1cf8e447e54d6e907ffa0e4681515f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:25.316001 7347 raft_consensus.cc:2932] T dd1cf8e447e54d6e907ffa0e4681515f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:25.322784 4300 catalog_manager.cc:5484] T dd1cf8e447e54d6e907ffa0e4681515f P d0456c5e55a542128688904e003d403c reported cstate change: config changed from index 5 to 6, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:25.371294 4837 consensus_queue.cc:237] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:25.373878 7483 raft_consensus.cc:1261] T 85fac458a4204f40a85d4ecba4718f58 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:29:25.374064 4771 consensus_peers.cc:485] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223): Couldn't send request to peer 0ddd486109f5434cace13d60d633ac04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 85fac458a4204f40a85d4ecba4718f58. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:25.374831 24006 consensus_queue.cc:1035] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:29:25.375033 7347 raft_consensus.cc:1261] T 85fac458a4204f40a85d4ecba4718f58 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:29:25.376499 24027 consensus_queue.cc:1035] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:29:25.395099 24027 raft_consensus.cc:2932] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:29:25.398763 7484 raft_consensus.cc:2932] T 85fac458a4204f40a85d4ecba4718f58 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:29:25.406112 4291 catalog_manager.cc:5484] T 85fac458a4204f40a85d4ecba4718f58 P 31a72286397c4edc9a63eca3412be168 reported cstate change: config changed from index -1 to 2, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:29:25.413544 7349 raft_consensus.cc:2932] T 85fac458a4204f40a85d4ecba4718f58 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0ddd486109f5434cace13d60d633ac04 (127.29.82.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:29:25.459458 4439 consensus_queue.cc:237] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:25.462090 7349 raft_consensus.cc:1261] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:25.462280 4703 raft_consensus.cc:1261] T 1c243b27429043aa8642d9b2ed595ab3 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Refusing update from remote peer 71e1375b799b439fa3c83ca763c039c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:25.462987 23826 consensus_queue.cc:1035] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:25.463424 24050 consensus_queue.cc:1035] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 21:29:25.465091 4375 consensus_peers.cc:485] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 -> Peer 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): Couldn't send request to peer 31a72286397c4edc9a63eca3412be168. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c243b27429043aa8642d9b2ed595ab3. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:25.484750 24050 raft_consensus.cc:2932] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:25.485265 4705 raft_consensus.cc:2932] T 1c243b27429043aa8642d9b2ed595ab3 P 398c3c8f3195492c900439ad5e1931bb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:25.494001 4291 catalog_manager.cc:5484] T 1c243b27429043aa8642d9b2ed595ab3 P 398c3c8f3195492c900439ad5e1931bb reported cstate change: config changed from index 5 to 6, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New cstate: current_term: 1 leader_uuid: "71e1375b799b439fa3c83ca763c039c1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:25.494251 7347 raft_consensus.cc:2932] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 31a72286397c4edc9a63eca3412be168 (127.29.82.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:25.519984 24279 ts_tablet_manager.cc:902] T a410b7da18004b23bdad650f59a30e8c P 73a1f2ca1f544de9b0e3b37015f85328: Initiating tablet copy from peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:29:25.537567 24279 tablet_copy_client.cc:308] T a410b7da18004b23bdad650f59a30e8c P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.69:33193
I20240502 21:29:25.538311 7236 tablet_copy_service.cc:140] P 39981a8dfe94462b9a73ba489af4c714: Received BeginTabletCopySession request for tablet a410b7da18004b23bdad650f59a30e8c from peer 73a1f2ca1f544de9b0e3b37015f85328 ({username='slave'} at 127.29.82.68:40041)
I20240502 21:29:25.538501 7236 tablet_copy_service.cc:161] P 39981a8dfe94462b9a73ba489af4c714: Beginning new tablet copy session on tablet a410b7da18004b23bdad650f59a30e8c from peer 73a1f2ca1f544de9b0e3b37015f85328 at {username='slave'} at 127.29.82.68:40041: session id = 73a1f2ca1f544de9b0e3b37015f85328-a410b7da18004b23bdad650f59a30e8c
I20240502 21:29:25.542150 7236 tablet_copy_source_session.cc:205] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:25.543440 24279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a410b7da18004b23bdad650f59a30e8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:25.550467 24279 tablet_copy_client.cc:783] T a410b7da18004b23bdad650f59a30e8c P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:25.550715 24279 tablet_copy_client.cc:647] T a410b7da18004b23bdad650f59a30e8c P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:25.552328 24279 tablet_copy_client.cc:523] T a410b7da18004b23bdad650f59a30e8c P 73a1f2ca1f544de9b0e3b37015f85328: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:25.555327 24279 tablet_bootstrap.cc:492] T a410b7da18004b23bdad650f59a30e8c P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap starting.
I20240502 21:29:25.563393 24279 tablet_bootstrap.cc:492] T a410b7da18004b23bdad650f59a30e8c P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:25.563748 24279 tablet_bootstrap.cc:492] T a410b7da18004b23bdad650f59a30e8c P 73a1f2ca1f544de9b0e3b37015f85328: Bootstrap complete.
I20240502 21:29:25.564009 24279 ts_tablet_manager.cc:1367] T a410b7da18004b23bdad650f59a30e8c P 73a1f2ca1f544de9b0e3b37015f85328: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.002s
I20240502 21:29:25.564705 24279 raft_consensus.cc:348] T a410b7da18004b23bdad650f59a30e8c P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:29:25.564944 24279 raft_consensus.cc:729] T a410b7da18004b23bdad650f59a30e8c P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Initialized, Role: FOLLOWER
I20240502 21:29:25.565155 24279 consensus_queue.cc:260] T a410b7da18004b23bdad650f59a30e8c P 73a1f2ca1f544de9b0e3b37015f85328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:29:25.565753 24279 ts_tablet_manager.cc:1398] T a410b7da18004b23bdad650f59a30e8c P 73a1f2ca1f544de9b0e3b37015f85328: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:29:25.566324 7236 tablet_copy_service.cc:342] P 39981a8dfe94462b9a73ba489af4c714: Request end of tablet copy session 73a1f2ca1f544de9b0e3b37015f85328-a410b7da18004b23bdad650f59a30e8c received from {username='slave'} at 127.29.82.68:40041
I20240502 21:29:25.566432 7236 tablet_copy_service.cc:433] P 39981a8dfe94462b9a73ba489af4c714: ending tablet copy session 73a1f2ca1f544de9b0e3b37015f85328-a410b7da18004b23bdad650f59a30e8c on tablet a410b7da18004b23bdad650f59a30e8c with peer 73a1f2ca1f544de9b0e3b37015f85328
I20240502 21:29:25.581099 7212 consensus_queue.cc:237] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:29:25.583890 4837 raft_consensus.cc:1261] T 5bd324f8962647ef9e8fc01a64efe69a P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:29:25.584002 7148 consensus_peers.cc:485] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5bd324f8962647ef9e8fc01a64efe69a. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:25.584400 24301 consensus_queue.cc:1035] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:29:25.584749 7618 raft_consensus.cc:1261] T 5bd324f8962647ef9e8fc01a64efe69a P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:29:25.585229 24301 consensus_queue.cc:1035] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:29:25.587829 24150 raft_consensus.cc:2932] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:29:25.588738 7618 raft_consensus.cc:2932] T 5bd324f8962647ef9e8fc01a64efe69a P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:29:25.597921 4837 raft_consensus.cc:2932] T 5bd324f8962647ef9e8fc01a64efe69a P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:29:25.607353 4300 catalog_manager.cc:5484] T 5bd324f8962647ef9e8fc01a64efe69a P 0ddd486109f5434cace13d60d633ac04 reported cstate change: config changed from index -1 to 2, VOTER 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) added. New cstate: current_term: 1 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:29:25.661983 4837 consensus_queue.cc:237] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
W20240502 21:29:25.664413 4772 consensus_peers.cc:485] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Couldn't send request to peer 39981a8dfe94462b9a73ba489af4c714. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86fa4255c6554ba28e141de3cb328976. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:25.664690 7618 raft_consensus.cc:1261] T 86fa4255c6554ba28e141de3cb328976 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:25.665324 23735 consensus_queue.cc:1035] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:25.665926 4573 raft_consensus.cc:1261] T 86fa4255c6554ba28e141de3cb328976 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:25.666457 23735 consensus_queue.cc:1035] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:25.670246 24027 raft_consensus.cc:2932] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:25.671393 24325 ts_tablet_manager.cc:902] T 2285212d78b0460ca329949eb3140e69 P 31a72286397c4edc9a63eca3412be168: Initiating tablet copy from peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:29:25.674909 4573 raft_consensus.cc:2932] T 86fa4255c6554ba28e141de3cb328976 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:25.680863 24325 tablet_copy_client.cc:308] T 2285212d78b0460ca329949eb3140e69 P 31a72286397c4edc9a63eca3412be168: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.67:34537
I20240502 21:29:25.682049 7618 raft_consensus.cc:2932] T 86fa4255c6554ba28e141de3cb328976 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:25.683748 4726 tablet_copy_service.cc:140] P 398c3c8f3195492c900439ad5e1931bb: Received BeginTabletCopySession request for tablet 2285212d78b0460ca329949eb3140e69 from peer 31a72286397c4edc9a63eca3412be168 ({username='slave'} at 127.29.82.71:44205)
I20240502 21:29:25.683627 4300 catalog_manager.cc:5484] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: config changed from index 5 to 6, VOTER 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69) added. New cstate: current_term: 1 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:25.683912 4726 tablet_copy_service.cc:161] P 398c3c8f3195492c900439ad5e1931bb: Beginning new tablet copy session on tablet 2285212d78b0460ca329949eb3140e69 from peer 31a72286397c4edc9a63eca3412be168 at {username='slave'} at 127.29.82.71:44205: session id = 31a72286397c4edc9a63eca3412be168-2285212d78b0460ca329949eb3140e69
I20240502 21:29:25.694499 4726 tablet_copy_source_session.cc:205] T 2285212d78b0460ca329949eb3140e69 P 398c3c8f3195492c900439ad5e1931bb: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:29:25.697656 24325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2285212d78b0460ca329949eb3140e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:25.703476 24325 tablet_copy_client.cc:783] T 2285212d78b0460ca329949eb3140e69 P 31a72286397c4edc9a63eca3412be168: tablet copy: Starting download of 5 data blocks...
I20240502 21:29:25.705737 24330 ts_tablet_manager.cc:902] T 686a74b8c1034112b293c4c7c6bcd99a P 71e1375b799b439fa3c83ca763c039c1: Initiating tablet copy from peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:29:25.711335 24325 tablet_copy_client.cc:647] T 2285212d78b0460ca329949eb3140e69 P 31a72286397c4edc9a63eca3412be168: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:25.713137 24325 tablet_copy_client.cc:523] T 2285212d78b0460ca329949eb3140e69 P 31a72286397c4edc9a63eca3412be168: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:25.716002 24325 tablet_bootstrap.cc:492] T 2285212d78b0460ca329949eb3140e69 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:29:25.730223 24330 tablet_copy_client.cc:308] T 686a74b8c1034112b293c4c7c6bcd99a P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.68:42449
I20240502 21:29:25.738427 4859 tablet_copy_service.cc:140] P 73a1f2ca1f544de9b0e3b37015f85328: Received BeginTabletCopySession request for tablet 686a74b8c1034112b293c4c7c6bcd99a from peer 71e1375b799b439fa3c83ca763c039c1 ({username='slave'} at 127.29.82.65:44725)
I20240502 21:29:25.738617 4859 tablet_copy_service.cc:161] P 73a1f2ca1f544de9b0e3b37015f85328: Beginning new tablet copy session on tablet 686a74b8c1034112b293c4c7c6bcd99a from peer 71e1375b799b439fa3c83ca763c039c1 at {username='slave'} at 127.29.82.65:44725: session id = 71e1375b799b439fa3c83ca763c039c1-686a74b8c1034112b293c4c7c6bcd99a
I20240502 21:29:25.742147 4859 tablet_copy_source_session.cc:205] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:25.743872 24330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 686a74b8c1034112b293c4c7c6bcd99a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:25.744814 24325 tablet_bootstrap.cc:492] T 2285212d78b0460ca329949eb3140e69 P 31a72286397c4edc9a63eca3412be168: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:25.745249 24325 tablet_bootstrap.cc:492] T 2285212d78b0460ca329949eb3140e69 P 31a72286397c4edc9a63eca3412be168: Bootstrap complete.
I20240502 21:29:25.745532 24325 ts_tablet_manager.cc:1367] T 2285212d78b0460ca329949eb3140e69 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.030s user 0.012s sys 0.006s
I20240502 21:29:25.746264 24325 raft_consensus.cc:348] T 2285212d78b0460ca329949eb3140e69 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:25.746542 24325 raft_consensus.cc:729] T 2285212d78b0460ca329949eb3140e69 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:29:25.746865 24325 consensus_queue.cc:260] T 2285212d78b0460ca329949eb3140e69 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:25.747499 24325 ts_tablet_manager.cc:1398] T 2285212d78b0460ca329949eb3140e69 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:29:25.748243 4726 tablet_copy_service.cc:342] P 398c3c8f3195492c900439ad5e1931bb: Request end of tablet copy session 31a72286397c4edc9a63eca3412be168-2285212d78b0460ca329949eb3140e69 received from {username='slave'} at 127.29.82.71:44205
I20240502 21:29:25.748371 4726 tablet_copy_service.cc:433] P 398c3c8f3195492c900439ad5e1931bb: ending tablet copy session 31a72286397c4edc9a63eca3412be168-2285212d78b0460ca329949eb3140e69 on tablet 2285212d78b0460ca329949eb3140e69 with peer 31a72286397c4edc9a63eca3412be168
I20240502 21:29:25.750742 24330 tablet_copy_client.cc:783] T 686a74b8c1034112b293c4c7c6bcd99a P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:25.750986 24330 tablet_copy_client.cc:647] T 686a74b8c1034112b293c4c7c6bcd99a P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:25.753450 24330 tablet_copy_client.cc:523] T 686a74b8c1034112b293c4c7c6bcd99a P 71e1375b799b439fa3c83ca763c039c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:25.757042 24330 tablet_bootstrap.cc:492] T 686a74b8c1034112b293c4c7c6bcd99a P 71e1375b799b439fa3c83ca763c039c1: Bootstrap starting.
I20240502 21:29:25.767357 24330 tablet_bootstrap.cc:492] T 686a74b8c1034112b293c4c7c6bcd99a P 71e1375b799b439fa3c83ca763c039c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:25.767807 24330 tablet_bootstrap.cc:492] T 686a74b8c1034112b293c4c7c6bcd99a P 71e1375b799b439fa3c83ca763c039c1: Bootstrap complete.
I20240502 21:29:25.768061 24330 ts_tablet_manager.cc:1367] T 686a74b8c1034112b293c4c7c6bcd99a P 71e1375b799b439fa3c83ca763c039c1: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20240502 21:29:25.768837 24330 raft_consensus.cc:348] T 686a74b8c1034112b293c4c7c6bcd99a P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:29:25.769078 24330 raft_consensus.cc:729] T 686a74b8c1034112b293c4c7c6bcd99a P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e1375b799b439fa3c83ca763c039c1, State: Initialized, Role: FOLLOWER
I20240502 21:29:25.769317 24330 consensus_queue.cc:260] T 686a74b8c1034112b293c4c7c6bcd99a P 71e1375b799b439fa3c83ca763c039c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:29:25.770000 24330 ts_tablet_manager.cc:1398] T 686a74b8c1034112b293c4c7c6bcd99a P 71e1375b799b439fa3c83ca763c039c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:29:25.772001 4859 tablet_copy_service.cc:342] P 73a1f2ca1f544de9b0e3b37015f85328: Request end of tablet copy session 71e1375b799b439fa3c83ca763c039c1-686a74b8c1034112b293c4c7c6bcd99a received from {username='slave'} at 127.29.82.65:44725
I20240502 21:29:25.772115 4859 tablet_copy_service.cc:433] P 73a1f2ca1f544de9b0e3b37015f85328: ending tablet copy session 71e1375b799b439fa3c83ca763c039c1-686a74b8c1034112b293c4c7c6bcd99a on tablet 686a74b8c1034112b293c4c7c6bcd99a with peer 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:25.781461 7618 consensus_queue.cc:237] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:29:25.784600 4573 raft_consensus.cc:1261] T 6065bdba6e2840b8b368d8f92d1fa260 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:29:25.784793 7551 consensus_peers.cc:485] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 -> Peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Couldn't send request to peer d0456c5e55a542128688904e003d403c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6065bdba6e2840b8b368d8f92d1fa260. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:25.785157 24130 consensus_queue.cc:1035] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:25.785513 7213 raft_consensus.cc:1261] T 6065bdba6e2840b8b368d8f92d1fa260 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:25.787712 24130 consensus_queue.cc:1035] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:25.797700 24358 ts_tablet_manager.cc:902] T 85fac458a4204f40a85d4ecba4718f58 P 0ddd486109f5434cace13d60d633ac04: Initiating tablet copy from peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:29:25.799230 24358 tablet_copy_client.cc:308] T 85fac458a4204f40a85d4ecba4718f58 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.68:42449
I20240502 21:29:25.799875 4859 tablet_copy_service.cc:140] P 73a1f2ca1f544de9b0e3b37015f85328: Received BeginTabletCopySession request for tablet 85fac458a4204f40a85d4ecba4718f58 from peer 0ddd486109f5434cace13d60d633ac04 ({username='slave'} at 127.29.82.72:48273)
I20240502 21:29:25.800021 4859 tablet_copy_service.cc:161] P 73a1f2ca1f544de9b0e3b37015f85328: Beginning new tablet copy session on tablet 85fac458a4204f40a85d4ecba4718f58 from peer 0ddd486109f5434cace13d60d633ac04 at {username='slave'} at 127.29.82.72:48273: session id = 0ddd486109f5434cace13d60d633ac04-85fac458a4204f40a85d4ecba4718f58
I20240502 21:29:25.803010 4859 tablet_copy_source_session.cc:205] T 85fac458a4204f40a85d4ecba4718f58 P 73a1f2ca1f544de9b0e3b37015f85328: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:25.804282 24358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85fac458a4204f40a85d4ecba4718f58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:25.805063 24130 raft_consensus.cc:2932] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:29:25.808192 4573 raft_consensus.cc:2932] T 6065bdba6e2840b8b368d8f92d1fa260 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:29:25.812141 7212 raft_consensus.cc:2932] T 6065bdba6e2840b8b368d8f92d1fa260 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:29:25.813661 24358 tablet_copy_client.cc:783] T 85fac458a4204f40a85d4ecba4718f58 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:25.813891 24358 tablet_copy_client.cc:647] T 85fac458a4204f40a85d4ecba4718f58 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:25.816397 24363 ts_tablet_manager.cc:902] T c6459a8c391249529d2672ab6c62a8af P 398c3c8f3195492c900439ad5e1931bb: Initiating tablet copy from peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
F20240502 21:29:25.817489 24363 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0a5fdbfa16c946a38a90be15079bda0c.queue_time_us, tablet-download-pool-0a5fdbfa16c946a38a90be15079bda0c.queue_time_us) (tablet-download-pool-0a5fdbfa16c946a38a90be15079bda0c.run_cpu_time_us, tablet-download-pool-0a5fdbfa16c946a38a90be15079bda0c.run_cpu_time_us) (tablet-download-pool-0a5fdbfa16c946a38a90be15079bda0c.run_wall_time_us, tablet-download-pool-0a5fdbfa16c946a38a90be15079bda0c.run_wall_time_us) (tablet-download-pool-1d3be8c5eaac423db11b282739cd1d40.queue_time_us, tablet-download-pool-1d3be8c5eaac423db11b282739cd1d40.queue_time_us) (tablet-download-pool-1d3be8c5eaac423db11b282739cd1d40.run_cpu_time_us, tablet-download-pool-1d3be8c5eaac423db11b282739cd1d40.run_cpu_time_us) (tablet-download-pool-1d3be8c5eaac423db11b282739cd1d40.run_wall_time_us, tablet-download-pool-1d3be8c5eaac423db11b282739cd1d40.run_wall_time_us) (tablet-download-pool-25be332e341d4869b2165da105c17a8a.queue_time_us, tablet-download-pool-25be332e341d4869b2165da105c17a8a.queue_time_us) (tablet-download-pool-25be332e341d4869b2165da105c17a8a.run_cpu_time_us, tablet-download-pool-25be332e341d4869b2165da105c17a8a.run_cpu_time_us) (tablet-download-pool-25be332e341d4869b2165da105c17a8a.run_wall_time_us, tablet-download-pool-25be332e341d4869b2165da105c17a8a.run_wall_time_us) (tablet-download-pool-26a00b5700a54672b7389e87c2a4dfbd.queue_time_us, tablet-download-pool-26a00b5700a54672b7389e87c2a4dfbd.queue_time_us) (tablet-download-pool-26a00b5700a54672b7389e87c2a4dfbd.run_cpu_time_us, tablet-download-pool-26a00b5700a54672b7389e87c2a4dfbd.run_cpu_time_us) (tablet-download-pool-26a00b5700a54672b7389e87c2a4dfbd.run_wall_time_us, tablet-download-pool-26a00b5700a54672b7389e87c2a4dfbd.run_wall_time_us) (tablet-download-pool-2ae670e3d8ae49f2b8c7df65b45b179e.queue_time_us, tablet-download-pool-2ae670e3d8ae49f2b8c7df65b45b179e.queue_time_us) (tablet-download-pool-2ae670e3d8ae49f2b8c7df65b45b179e.run_cpu_time_us, tablet-download-pool-2ae670e3d8ae49f2b8c7df65b45b179e.run_cpu_time_us) (tablet-download-pool-2ae670e3d8ae49f2b8c7df65b45b179e.run_wall_time_us, tablet-download-pool-2ae670e3d8ae49f2b8c7df65b45b179e.run_wall_time_us) (tablet-download-pool-42edb2b3ff094c71872fde7949b61b3b.queue_time_us, tablet-download-pool-42edb2b3ff094c71872fde7949b61b3b.queue_time_us) (tablet-download-pool-42edb2b3ff094c71872fde7949b61b3b.run_cpu_time_us, tablet-download-pool-42edb2b3ff094c71872fde7949b61b3b.run_cpu_time_us) (tablet-download-pool-42edb2b3ff094c71872fde7949b61b3b.run_wall_time_us, tablet-download-pool-42edb2b3ff094c71872fde7949b61b3b.run_wall_time_us) (tablet-download-pool-686a74b8c1034112b293c4c7c6bcd99a.queue_time_us, tablet-download-pool-686a74b8c1034112b293c4c7c6bcd99a.queue_time_us) (tablet-download-pool-686a74b8c1034112b293c4c7c6bcd99a.run_cpu_time_us, tablet-download-pool-686a74b8c1034112b293c4c7c6bcd99a.run_cpu_time_us) (tablet-download-pool-686a74b8c1034112b293c4c7c6bcd99a.run_wall_time_us, tablet-download-pool-686a74b8c1034112b293c4c7c6bcd99a.run_wall_time_us) (tablet-download-pool-6d0334be884c45c3bd9e49095454dfb1.queue_time_us, tablet-download-pool-6d0334be884c45c3bd9e49095454dfb1.queue_time_us) (tablet-download-pool-6d0334be884c45c3bd9e49095454dfb1.run_cpu_time_us, tablet-download-pool-6d0334be884c45c3bd9e49095454dfb1.run_cpu_time_us) (tablet-download-pool-6d0334be884c45c3bd9e49095454dfb1.run_wall_time_us, tablet-download-pool-6d0334be884c45c3bd9e49095454dfb1.run_wall_time_us) (tablet-download-pool-7b5db855b5b24c8b87d65c7973f90bbb.queue_time_us, tablet-download-pool-7b5db855b5b24c8b87d65c7973f90bbb.queue_time_us) (tablet-download-pool-7b5db855b5b24c8b87d65c7973f90bbb.run_cpu_time_us, tablet-download-pool-7b5db855b5b24c8b87d65c7973f90bbb.run_cpu_time_us) (tablet-download-pool-7b5db855b5b24c8b87d65c7973f90bbb.run_wall_time_us, tablet-download-pool-7b5db855b5b24c8b87d65c7973f90bbb.run_wall_time_us) (tablet-download-pool-80304ab48274408393c2e245477c3a78.queue_time_us, tablet-download-pool-80304ab48274408393c2e245477c3a78.queue_time_us) (tablet-download-pool-80304ab48274408393c2e245477c3a78.run_cpu_time_us, tablet-download-pool-80304ab48274408393c2e245477c3a78.run_cpu_time_us) (tablet-download-pool-80304ab48274408393c2e245477c3a78.run_wall_time_us, tablet-download-pool-80304ab48274408393c2e245477c3a78.run_wall_time_us) (tablet-download-pool-81db5e77d00446cba3eb8345ee3df476.queue_time_us, tablet-download-pool-81db5e77d00446cba3eb8345ee3df476.queue_time_us) (tablet-download-pool-81db5e77d00446cba3eb8345ee3df476.run_cpu_time_us, tablet-download-pool-81db5e77d00446cba3eb8345ee3df476.run_cpu_time_us) (tablet-download-pool-81db5e77d00446cba3eb8345ee3df476.run_wall_time_us, tablet-download-pool-81db5e77d00446cba3eb8345ee3df476.run_wall_time_us) (tablet-download-pool-8991236419e642bfa438b9e45a1b8235.queue_time_us, tablet-download-pool-8991236419e642bfa438b9e45a1b8235.queue_time_us) (tablet-download-pool-8991236419e642bfa438b9e45a1b8235.run_cpu_time_us, tablet-download-pool-8991236419e642bfa438b9e45a1b8235.run_cpu_time_us) (tablet-download-pool-8991236419e642bfa438b9e45a1b8235.run_wall_time_us, tablet-download-pool-8991236419e642bfa438b9e45a1b8235.run_wall_time_us) (tablet-download-pool-94e66998ede944a6b00478270029ebcf.queue_time_us, tablet-download-pool-94e66998ede944a6b00478270029ebcf.queue_time_us) (tablet-download-pool-94e66998ede944a6b00478270029ebcf.run_cpu_time_us, tablet-download-pool-94e66998ede944a6b00478270029ebcf.run_cpu_time_us) (tablet-download-pool-94e66998ede944a6b00478270029ebcf.run_wall_time_us, tablet-download-pool-94e66998ede944a6b00478270029ebcf.run_wall_time_us) (tablet-download-pool-c6459a8c391249529d2672ab6c62a8af.queue_time_us, tablet-download-pool-c6459a8c391249529d2672ab6c62a8af.queue_time_us) (tablet-download-pool-ed822833ab3c451f96d722980b24c2b0.queue_time_us, tablet-download-pool-ed822833ab3c451f96d722980b24c2b0.queue_time_us) (tablet-download-pool-ed822833ab3c451f96d722980b24c2b0.run_cpu_time_us, tablet-download-pool-ed822833ab3c451f96d722980b24c2b0.run_cpu_time_us) (tablet-download-pool-ed822833ab3c451f96d722980b24c2b0.run_wall_time_us, tablet-download-pool-ed822833ab3c451f96d722980b24c2b0.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240502 21:29:25.820070 4291 catalog_manager.cc:5484] T 6065bdba6e2840b8b368d8f92d1fa260 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: config changed from index 3 to 4, VOTER d0456c5e55a542128688904e003d403c (127.29.82.70) added. New cstate: current_term: 1 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
@ 0x7fa8b8733dcd google::LogMessage::Fail() at ??:0
@ 0x7fa8b8737b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fa8b87337cc google::LogMessage::Flush() at ??:0
@ 0x7fa8b8734f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 21:29:25.839121 24358 tablet_copy_client.cc:523] T 85fac458a4204f40a85d4ecba4718f58 P 0ddd486109f5434cace13d60d633ac04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:25.842864 24358 tablet_bootstrap.cc:492] T 85fac458a4204f40a85d4ecba4718f58 P 0ddd486109f5434cace13d60d633ac04: Bootstrap starting.
I20240502 21:29:25.845077 7531 maintenance_manager.cc:419] P 31a72286397c4edc9a63eca3412be168: Scheduling UndoDeltaBlockGCOp(2285212d78b0460ca329949eb3140e69): 4925 bytes on disk
@ 0x7fa8bb474742 kudu::TraceMetrics::InternName() at ??:0
I20240502 21:29:25.845750 7421 maintenance_manager.cc:643] P 31a72286397c4edc9a63eca3412be168: UndoDeltaBlockGCOp(2285212d78b0460ca329949eb3140e69) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
@ 0x7fa8bb440f8b kudu::ThreadPool::ThreadPool() at ??:0
I20240502 21:29:25.857692 24358 tablet_bootstrap.cc:492] T 85fac458a4204f40a85d4ecba4718f58 P 0ddd486109f5434cace13d60d633ac04: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:25.858052 24358 tablet_bootstrap.cc:492] T 85fac458a4204f40a85d4ecba4718f58 P 0ddd486109f5434cace13d60d633ac04: Bootstrap complete.
I20240502 21:29:25.858291 24358 ts_tablet_manager.cc:1367] T 85fac458a4204f40a85d4ecba4718f58 P 0ddd486109f5434cace13d60d633ac04: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.004s
I20240502 21:29:25.859058 24358 raft_consensus.cc:348] T 85fac458a4204f40a85d4ecba4718f58 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:25.859984 24358 raft_consensus.cc:729] T 85fac458a4204f40a85d4ecba4718f58 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Initialized, Role: FOLLOWER
I20240502 21:29:25.860244 24358 consensus_queue.cc:260] T 85fac458a4204f40a85d4ecba4718f58 P 0ddd486109f5434cace13d60d633ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:25.875259 24358 ts_tablet_manager.cc:1398] T 85fac458a4204f40a85d4ecba4718f58 P 0ddd486109f5434cace13d60d633ac04: Time spent starting tablet: real 0.017s user 0.003s sys 0.001s
I20240502 21:29:25.875916 4859 tablet_copy_service.cc:342] P 73a1f2ca1f544de9b0e3b37015f85328: Request end of tablet copy session 0ddd486109f5434cace13d60d633ac04-85fac458a4204f40a85d4ecba4718f58 received from {username='slave'} at 127.29.82.72:48273
I20240502 21:29:25.876031 4859 tablet_copy_service.cc:433] P 73a1f2ca1f544de9b0e3b37015f85328: ending tablet copy session 0ddd486109f5434cace13d60d633ac04-85fac458a4204f40a85d4ecba4718f58 on tablet 85fac458a4204f40a85d4ecba4718f58 with peer 0ddd486109f5434cace13d60d633ac04
@ 0x7fa8bb43acd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fa8ca88d8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 21:29:25.890266 24372 ts_tablet_manager.cc:902] T dd1cf8e447e54d6e907ffa0e4681515f P 39981a8dfe94462b9a73ba489af4c714: Initiating tablet copy from peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
@ 0x7fa8ca8a5daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fa8ca9d0480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fa8caa08a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fa8ca9eb981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7fa8ca9eb70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fa8bb444b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa8bb44819c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa8bb448019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fa8bb42301e kudu::Thread::SuperviseThread() at ??:0
I20240502 21:29:25.942550 24372 tablet_copy_client.cc:308] T dd1cf8e447e54d6e907ffa0e4681515f P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.68:42449
I20240502 21:29:25.943163 4859 tablet_copy_service.cc:140] P 73a1f2ca1f544de9b0e3b37015f85328: Received BeginTabletCopySession request for tablet dd1cf8e447e54d6e907ffa0e4681515f from peer 39981a8dfe94462b9a73ba489af4c714 ({username='slave'} at 127.29.82.69:57807)
I20240502 21:29:25.943297 4859 tablet_copy_service.cc:161] P 73a1f2ca1f544de9b0e3b37015f85328: Beginning new tablet copy session on tablet dd1cf8e447e54d6e907ffa0e4681515f from peer 39981a8dfe94462b9a73ba489af4c714 at {username='slave'} at 127.29.82.69:57807: session id = 39981a8dfe94462b9a73ba489af4c714-dd1cf8e447e54d6e907ffa0e4681515f
I20240502 21:29:25.946485 4859 tablet_copy_source_session.cc:205] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328: Tablet Copy: opened 5 blocks and 1 log segments
@ 0x7fa8be7a86db start_thread at ??:0
I20240502 21:29:25.949744 24372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd1cf8e447e54d6e907ffa0e4681515f. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fa8b75e671f clone at ??:0
I20240502 21:29:25.953280 24325 ts_tablet_manager.cc:902] T 1c243b27429043aa8642d9b2ed595ab3 P 31a72286397c4edc9a63eca3412be168: Initiating tablet copy from peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:29:25.954685 24325 tablet_copy_client.cc:308] T 1c243b27429043aa8642d9b2ed595ab3 P 31a72286397c4edc9a63eca3412be168: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.65:42781
I20240502 21:29:25.955296 4463 tablet_copy_service.cc:140] P 71e1375b799b439fa3c83ca763c039c1: Received BeginTabletCopySession request for tablet 1c243b27429043aa8642d9b2ed595ab3 from peer 31a72286397c4edc9a63eca3412be168 ({username='slave'} at 127.29.82.71:41363)
I20240502 21:29:25.955469 4463 tablet_copy_service.cc:161] P 71e1375b799b439fa3c83ca763c039c1: Beginning new tablet copy session on tablet 1c243b27429043aa8642d9b2ed595ab3 from peer 31a72286397c4edc9a63eca3412be168 at {username='slave'} at 127.29.82.71:41363: session id = 31a72286397c4edc9a63eca3412be168-1c243b27429043aa8642d9b2ed595ab3
I20240502 21:29:25.958667 4463 tablet_copy_source_session.cc:205] T 1c243b27429043aa8642d9b2ed595ab3 P 71e1375b799b439fa3c83ca763c039c1: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:29:25.961377 24372 tablet_copy_client.cc:783] T dd1cf8e447e54d6e907ffa0e4681515f P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Starting download of 5 data blocks...
I20240502 21:29:25.962327 24325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c243b27429043aa8642d9b2ed595ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:25.975551 24325 tablet_copy_client.cc:783] T 1c243b27429043aa8642d9b2ed595ab3 P 31a72286397c4edc9a63eca3412be168: tablet copy: Starting download of 5 data blocks...
I20240502 21:29:25.977265 24372 tablet_copy_client.cc:647] T dd1cf8e447e54d6e907ffa0e4681515f P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:25.980057 24372 tablet_copy_client.cc:523] T dd1cf8e447e54d6e907ffa0e4681515f P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:25.983834 24325 tablet_copy_client.cc:647] T 1c243b27429043aa8642d9b2ed595ab3 P 31a72286397c4edc9a63eca3412be168: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:25.984527 24372 tablet_bootstrap.cc:492] T dd1cf8e447e54d6e907ffa0e4681515f P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:29:25.992154 24325 tablet_copy_client.cc:523] T 1c243b27429043aa8642d9b2ed595ab3 P 31a72286397c4edc9a63eca3412be168: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:26.001068 24325 tablet_bootstrap.cc:492] T 1c243b27429043aa8642d9b2ed595ab3 P 31a72286397c4edc9a63eca3412be168: Bootstrap starting.
I20240502 21:29:26.030514 4838 raft_consensus.cc:1203] T a410b7da18004b23bdad650f59a30e8c P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:29:26.046485 24372 tablet_bootstrap.cc:492] T dd1cf8e447e54d6e907ffa0e4681515f P 39981a8dfe94462b9a73ba489af4c714: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:26.046880 24372 tablet_bootstrap.cc:492] T dd1cf8e447e54d6e907ffa0e4681515f P 39981a8dfe94462b9a73ba489af4c714: Bootstrap complete.
I20240502 21:29:26.047166 24372 ts_tablet_manager.cc:1367] T dd1cf8e447e54d6e907ffa0e4681515f P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.063s user 0.034s sys 0.005s
I20240502 21:29:26.047971 24372 raft_consensus.cc:348] T dd1cf8e447e54d6e907ffa0e4681515f P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:26.048211 24372 raft_consensus.cc:729] T dd1cf8e447e54d6e907ffa0e4681515f P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:29:26.048491 24372 consensus_queue.cc:260] T dd1cf8e447e54d6e907ffa0e4681515f P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:26.050014 24372 ts_tablet_manager.cc:1398] T dd1cf8e447e54d6e907ffa0e4681515f P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20240502 21:29:26.052356 4859 tablet_copy_service.cc:342] P 73a1f2ca1f544de9b0e3b37015f85328: Request end of tablet copy session 39981a8dfe94462b9a73ba489af4c714-dd1cf8e447e54d6e907ffa0e4681515f received from {username='slave'} at 127.29.82.69:57807
I20240502 21:29:26.052457 4859 tablet_copy_service.cc:433] P 73a1f2ca1f544de9b0e3b37015f85328: ending tablet copy session 39981a8dfe94462b9a73ba489af4c714-dd1cf8e447e54d6e907ffa0e4681515f on tablet dd1cf8e447e54d6e907ffa0e4681515f with peer 39981a8dfe94462b9a73ba489af4c714
I20240502 21:29:26.061259 24325 tablet_bootstrap.cc:492] T 1c243b27429043aa8642d9b2ed595ab3 P 31a72286397c4edc9a63eca3412be168: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:26.061674 24325 tablet_bootstrap.cc:492] T 1c243b27429043aa8642d9b2ed595ab3 P 31a72286397c4edc9a63eca3412be168: Bootstrap complete.
I20240502 21:29:26.061941 24325 ts_tablet_manager.cc:1367] T 1c243b27429043aa8642d9b2ed595ab3 P 31a72286397c4edc9a63eca3412be168: Time spent bootstrapping tablet: real 0.061s user 0.014s sys 0.008s
I20240502 21:29:26.063068 24325 raft_consensus.cc:348] T 1c243b27429043aa8642d9b2ed595ab3 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:26.063352 24325 raft_consensus.cc:729] T 1c243b27429043aa8642d9b2ed595ab3 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Initialized, Role: FOLLOWER
I20240502 21:29:26.063784 24325 consensus_queue.cc:260] T 1c243b27429043aa8642d9b2ed595ab3 P 31a72286397c4edc9a63eca3412be168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:26.064520 24325 ts_tablet_manager.cc:1398] T 1c243b27429043aa8642d9b2ed595ab3 P 31a72286397c4edc9a63eca3412be168: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 21:29:26.070676 4463 tablet_copy_service.cc:342] P 71e1375b799b439fa3c83ca763c039c1: Request end of tablet copy session 31a72286397c4edc9a63eca3412be168-1c243b27429043aa8642d9b2ed595ab3 received from {username='slave'} at 127.29.82.71:41363
I20240502 21:29:26.070842 4463 tablet_copy_service.cc:433] P 71e1375b799b439fa3c83ca763c039c1: ending tablet copy session 31a72286397c4edc9a63eca3412be168-1c243b27429043aa8642d9b2ed595ab3 on tablet 1c243b27429043aa8642d9b2ed595ab3 with peer 31a72286397c4edc9a63eca3412be168
I20240502 21:29:26.101614 7531 maintenance_manager.cc:419] P 31a72286397c4edc9a63eca3412be168: Scheduling UndoDeltaBlockGCOp(1c243b27429043aa8642d9b2ed595ab3): 4841 bytes on disk
I20240502 21:29:26.102519 7421 maintenance_manager.cc:643] P 31a72286397c4edc9a63eca3412be168: UndoDeltaBlockGCOp(1c243b27429043aa8642d9b2ed595ab3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240502 21:29:26.142956 4439 raft_consensus.cc:1203] T 686a74b8c1034112b293c4c7c6bcd99a P 71e1375b799b439fa3c83ca763c039c1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:29:26.167112 24330 ts_tablet_manager.cc:902] T 5bd324f8962647ef9e8fc01a64efe69a P 71e1375b799b439fa3c83ca763c039c1: Initiating tablet copy from peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
F20240502 21:29:26.167443 24330 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0c56940914cb4161bd258bf881f06c07.queue_time_us, tablet-download-pool-0c56940914cb4161bd258bf881f06c07.queue_time_us) (tablet-download-pool-0c56940914cb4161bd258bf881f06c07.run_cpu_time_us, tablet-download-pool-0c56940914cb4161bd258bf881f06c07.run_cpu_time_us) (tablet-download-pool-0c56940914cb4161bd258bf881f06c07.run_wall_time_us, tablet-download-pool-0c56940914cb4161bd258bf881f06c07.run_wall_time_us) (tablet-download-pool-202f8b10d5674afa875da2f3929bd8da.queue_time_us, tablet-download-pool-202f8b10d5674afa875da2f3929bd8da.queue_time_us) (tablet-download-pool-202f8b10d5674afa875da2f3929bd8da.run_cpu_time_us, tablet-download-pool-202f8b10d5674afa875da2f3929bd8da.run_cpu_time_us) (tablet-download-pool-202f8b10d5674afa875da2f3929bd8da.run_wall_time_us, tablet-download-pool-202f8b10d5674afa875da2f3929bd8da.run_wall_time_us) (tablet-download-pool-269eb0e719804925961e4bcd17c18596.queue_time_us, tablet-download-pool-269eb0e719804925961e4bcd17c18596.queue_time_us) (tablet-download-pool-269eb0e719804925961e4bcd17c18596.run_cpu_time_us, tablet-download-pool-269eb0e719804925961e4bcd17c18596.run_cpu_time_us) (tablet-download-pool-269eb0e719804925961e4bcd17c18596.run_wall_time_us, tablet-download-pool-269eb0e719804925961e4bcd17c18596.run_wall_time_us) (tablet-download-pool-357690713cd44eb3b2a297bfd6bbc17d.queue_time_us, tablet-download-pool-357690713cd44eb3b2a297bfd6bbc17d.queue_time_us) (tablet-download-pool-357690713cd44eb3b2a297bfd6bbc17d.run_cpu_time_us, tablet-download-pool-357690713cd44eb3b2a297bfd6bbc17d.run_cpu_time_us) (tablet-download-pool-357690713cd44eb3b2a297bfd6bbc17d.run_wall_time_us, tablet-download-pool-357690713cd44eb3b2a297bfd6bbc17d.run_wall_time_us) (tablet-download-pool-4c06f3bc97254bff8181d3f6f02470b4.queue_time_us, tablet-download-pool-4c06f3bc97254bff8181d3f6f02470b4.queue_time_us) (tablet-download-pool-4c06f3bc97254bff8181d3f6f02470b4.run_cpu_time_us, tablet-download-pool-4c06f3bc97254bff8181d3f6f02470b4.run_cpu_time_us) (tablet-download-pool-4c06f3bc97254bff8181d3f6f02470b4.run_wall_time_us, tablet-download-pool-4c06f3bc97254bff8181d3f6f02470b4.run_wall_time_us) (tablet-download-pool-5bd324f8962647ef9e8fc01a64efe69a.queue_time_us, tablet-download-pool-5bd324f8962647ef9e8fc01a64efe69a.queue_time_us) (tablet-download-pool-61e18d2b13644f6a99225c106c594914.queue_time_us, tablet-download-pool-61e18d2b13644f6a99225c106c594914.queue_time_us) (tablet-download-pool-61e18d2b13644f6a99225c106c594914.run_cpu_time_us, tablet-download-pool-61e18d2b13644f6a99225c106c594914.run_cpu_time_us) (tablet-download-pool-61e18d2b13644f6a99225c106c594914.run_wall_time_us, tablet-download-pool-61e18d2b13644f6a99225c106c594914.run_wall_time_us) (tablet-download-pool-686a74b8c1034112b293c4c7c6bcd99a.queue_time_us, tablet-download-pool-686a74b8c1034112b293c4c7c6bcd99a.queue_time_us) (tablet-download-pool-686a74b8c1034112b293c4c7c6bcd99a.run_cpu_time_us, tablet-download-pool-686a74b8c1034112b293c4c7c6bcd99a.run_cpu_time_us) (tablet-download-pool-686a74b8c1034112b293c4c7c6bcd99a.run_wall_time_us, tablet-download-pool-686a74b8c1034112b293c4c7c6bcd99a.run_wall_time_us) (tablet-download-pool-845233761c2e45b0a23e25107f5461f1.queue_time_us, tablet-download-pool-845233761c2e45b0a23e25107f5461f1.queue_time_us) (tablet-download-pool-845233761c2e45b0a23e25107f5461f1.run_cpu_time_us, tablet-download-pool-845233761c2e45b0a23e25107f5461f1.run_cpu_time_us) (tablet-download-pool-845233761c2e45b0a23e25107f5461f1.run_wall_time_us, tablet-download-pool-845233761c2e45b0a23e25107f5461f1.run_wall_time_us) (tablet-download-pool-84c8bdb300064adc86a4b23e07d512d3.queue_time_us, tablet-download-pool-84c8bdb300064adc86a4b23e07d512d3.queue_time_us) (tablet-download-pool-84c8bdb300064adc86a4b23e07d512d3.run_cpu_time_us, tablet-download-pool-84c8bdb300064adc86a4b23e07d512d3.run_cpu_time_us) (tablet-download-pool-84c8bdb300064adc86a4b23e07d512d3.run_wall_time_us, tablet-download-pool-84c8bdb300064adc86a4b23e07d512d3.run_wall_time_us) (tablet-download-pool-9e5434cb329e4bea85317d18f6e6867d.queue_time_us, tablet-download-pool-9e5434cb329e4bea85317d18f6e6867d.queue_time_us) (tablet-download-pool-9e5434cb329e4bea85317d18f6e6867d.run_cpu_time_us, tablet-download-pool-9e5434cb329e4bea85317d18f6e6867d.run_cpu_time_us) (tablet-download-pool-9e5434cb329e4bea85317d18f6e6867d.run_wall_time_us, tablet-download-pool-9e5434cb329e4bea85317d18f6e6867d.run_wall_time_us) (tablet-download-pool-d095a5e8f1f34b5ea2b6797ef06f79c6.queue_time_us, tablet-download-pool-d095a5e8f1f34b5ea2b6797ef06f79c6.queue_time_us) (tablet-download-pool-d095a5e8f1f34b5ea2b6797ef06f79c6.run_cpu_time_us, tablet-download-pool-d095a5e8f1f34b5ea2b6797ef06f79c6.run_cpu_time_us) (tablet-download-pool-d095a5e8f1f34b5ea2b6797ef06f79c6.run_wall_time_us, tablet-download-pool-d095a5e8f1f34b5ea2b6797ef06f79c6.run_wall_time_us) (tablet-download-pool-d09a80f79473469e92310ded0f4edfdd.queue_time_us, tablet-download-pool-d09a80f79473469e92310ded0f4edfdd.queue_time_us) (tablet-download-pool-d09a80f79473469e92310ded0f4edfdd.run_cpu_time_us, tablet-download-pool-d09a80f79473469e92310ded0f4edfdd.run_cpu_time_us) (tablet-download-pool-d09a80f79473469e92310ded0f4edfdd.run_wall_time_us, tablet-download-pool-d09a80f79473469e92310ded0f4edfdd.run_wall_time_us) (tablet-download-pool-e8f42cdee5c24db982392d2057b588a2.queue_time_us, tablet-download-pool-e8f42cdee5c24db982392d2057b588a2.queue_time_us) (tablet-download-pool-e8f42cdee5c24db982392d2057b588a2.run_cpu_time_us, tablet-download-pool-e8f42cdee5c24db982392d2057b588a2.run_cpu_time_us) (tablet-download-pool-e8f42cdee5c24db982392d2057b588a2.run_wall_time_us, tablet-download-pool-e8f42cdee5c24db982392d2057b588a2.run_wall_time_us) (tablet-download-pool-fd610e340cd3409397d8f23b623b4074.queue_time_us, tablet-download-pool-fd610e340cd3409397d8f23b623b4074.queue_time_us) (tablet-download-pool-fd610e340cd3409397d8f23b623b4074.run_cpu_time_us, tablet-download-pool-fd610e340cd3409397d8f23b623b4074.run_cpu_time_us) (tablet-download-pool-fd610e340cd3409397d8f23b623b4074.run_wall_time_us, tablet-download-pool-fd610e340cd3409397d8f23b623b4074.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f0392ec9dcd google::LogMessage::Fail() at ??:0
@ 0x7f0392ecdb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f0392ec97cc google::LogMessage::Flush() at ??:0
@ 0x7f0392ecaf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0395c0a742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f0395bd6f8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f0395bd0cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f03a50238de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f03a503bdaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f03a5166480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f03a519ea8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f03a5181981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f03a518170e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f0395bdab75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0395bde19c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f0395bde019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240502 21:29:26.233909 7617 raft_consensus.cc:1203] T 85fac458a4204f40a85d4ecba4718f58 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
@ 0x7f0395bb901e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0398f3e6db start_thread at ??:0
@ 0x7f0391d7c71f clone at ??:0
I20240502 21:29:26.246667 24410 ts_tablet_manager.cc:902] T 6065bdba6e2840b8b368d8f92d1fa260 P d0456c5e55a542128688904e003d403c: Initiating tablet copy from peer 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:29:26.254434 24372 ts_tablet_manager.cc:902] T 86fa4255c6554ba28e141de3cb328976 P 39981a8dfe94462b9a73ba489af4c714: Initiating tablet copy from peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:29:26.265429 7262 maintenance_manager.cc:419] P 39981a8dfe94462b9a73ba489af4c714: Scheduling UndoDeltaBlockGCOp(dd1cf8e447e54d6e907ffa0e4681515f): 4586 bytes on disk
I20240502 21:29:26.266109 7152 maintenance_manager.cc:643] P 39981a8dfe94462b9a73ba489af4c714: UndoDeltaBlockGCOp(dd1cf8e447e54d6e907ffa0e4681515f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
I20240502 21:29:26.291664 24372 tablet_copy_client.cc:308] T 86fa4255c6554ba28e141de3cb328976 P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.68:42449
I20240502 21:29:26.292526 4859 tablet_copy_service.cc:140] P 73a1f2ca1f544de9b0e3b37015f85328: Received BeginTabletCopySession request for tablet 86fa4255c6554ba28e141de3cb328976 from peer 39981a8dfe94462b9a73ba489af4c714 ({username='slave'} at 127.29.82.69:57807)
I20240502 21:29:26.292698 4859 tablet_copy_service.cc:161] P 73a1f2ca1f544de9b0e3b37015f85328: Beginning new tablet copy session on tablet 86fa4255c6554ba28e141de3cb328976 from peer 39981a8dfe94462b9a73ba489af4c714 at {username='slave'} at 127.29.82.69:57807: session id = 39981a8dfe94462b9a73ba489af4c714-86fa4255c6554ba28e141de3cb328976
I20240502 21:29:26.294340 24410 tablet_copy_client.cc:308] T 6065bdba6e2840b8b368d8f92d1fa260 P d0456c5e55a542128688904e003d403c: tablet copy: Beginning tablet copy session from remote peer at address 127.29.82.72:41223
I20240502 21:29:26.295193 7638 tablet_copy_service.cc:140] P 0ddd486109f5434cace13d60d633ac04: Received BeginTabletCopySession request for tablet 6065bdba6e2840b8b368d8f92d1fa260 from peer d0456c5e55a542128688904e003d403c ({username='slave'} at 127.29.82.70:37961)
I20240502 21:29:26.295364 7638 tablet_copy_service.cc:161] P 0ddd486109f5434cace13d60d633ac04: Beginning new tablet copy session on tablet 6065bdba6e2840b8b368d8f92d1fa260 from peer d0456c5e55a542128688904e003d403c at {username='slave'} at 127.29.82.70:37961: session id = d0456c5e55a542128688904e003d403c-6065bdba6e2840b8b368d8f92d1fa260
I20240502 21:29:26.296227 4859 tablet_copy_source_session.cc:205] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:29:26.299469 24372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86fa4255c6554ba28e141de3cb328976. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:26.308002 7638 tablet_copy_source_session.cc:205] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:26.309438 24410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6065bdba6e2840b8b368d8f92d1fa260. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:26.309980 24372 tablet_copy_client.cc:783] T 86fa4255c6554ba28e141de3cb328976 P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Starting download of 5 data blocks...
I20240502 21:29:26.320616 24410 tablet_copy_client.cc:783] T 6065bdba6e2840b8b368d8f92d1fa260 P d0456c5e55a542128688904e003d403c: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:26.320876 24410 tablet_copy_client.cc:647] T 6065bdba6e2840b8b368d8f92d1fa260 P d0456c5e55a542128688904e003d403c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:26.322614 24372 tablet_copy_client.cc:647] T 86fa4255c6554ba28e141de3cb328976 P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:26.325994 24372 tablet_copy_client.cc:523] T 86fa4255c6554ba28e141de3cb328976 P 39981a8dfe94462b9a73ba489af4c714: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:26.332110 24372 tablet_bootstrap.cc:492] T 86fa4255c6554ba28e141de3cb328976 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap starting.
I20240502 21:29:26.334916 24410 tablet_copy_client.cc:523] T 6065bdba6e2840b8b368d8f92d1fa260 P d0456c5e55a542128688904e003d403c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:26.342355 24410 tablet_bootstrap.cc:492] T 6065bdba6e2840b8b368d8f92d1fa260 P d0456c5e55a542128688904e003d403c: Bootstrap starting.
W20240502 21:29:26.353822 7151 connection.cc:535] client connection to 127.29.82.67:34537 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:29:26.353876 7285 connection.cc:535] client connection to 127.29.82.67:34537 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:29:26.354095 7151 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:29:26.354104 7285 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:29:26.359310 24410 tablet_bootstrap.cc:492] T 6065bdba6e2840b8b368d8f92d1fa260 P d0456c5e55a542128688904e003d403c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:26.359934 24410 tablet_bootstrap.cc:492] T 6065bdba6e2840b8b368d8f92d1fa260 P d0456c5e55a542128688904e003d403c: Bootstrap complete.
W20240502 21:29:26.360723 7552 connection.cc:535] client connection to 127.29.82.67:34537 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:29:26.360713 4511 connection.cc:535] client connection to 127.29.82.67:34537 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:29:26.360931 4511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:29:26.360954 7552 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:29:26.360260 24410 ts_tablet_manager.cc:1367] T 6065bdba6e2840b8b368d8f92d1fa260 P d0456c5e55a542128688904e003d403c: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.003s
I20240502 21:29:26.370195 24410 raft_consensus.cc:348] T 6065bdba6e2840b8b368d8f92d1fa260 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
W20240502 21:29:26.370383 7420 connection.cc:535] client connection to 127.29.82.67:34537 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:29:26.370447 24410 raft_consensus.cc:729] T 6065bdba6e2840b8b368d8f92d1fa260 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0456c5e55a542128688904e003d403c, State: Initialized, Role: FOLLOWER
W20240502 21:29:26.370620 7420 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:29:26.371237 24372 tablet_bootstrap.cc:492] T 86fa4255c6554ba28e141de3cb328976 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:26.372797 24372 tablet_bootstrap.cc:492] T 86fa4255c6554ba28e141de3cb328976 P 39981a8dfe94462b9a73ba489af4c714: Bootstrap complete.
I20240502 21:29:26.373093 24372 ts_tablet_manager.cc:1367] T 86fa4255c6554ba28e141de3cb328976 P 39981a8dfe94462b9a73ba489af4c714: Time spent bootstrapping tablet: real 0.041s user 0.020s sys 0.000s
I20240502 21:29:26.370669 24410 consensus_queue.cc:260] T 6065bdba6e2840b8b368d8f92d1fa260 P d0456c5e55a542128688904e003d403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:29:26.377774 24410 ts_tablet_manager.cc:1398] T 6065bdba6e2840b8b368d8f92d1fa260 P d0456c5e55a542128688904e003d403c: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240502 21:29:26.377642 24372 raft_consensus.cc:348] T 86fa4255c6554ba28e141de3cb328976 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:26.378464 24372 raft_consensus.cc:729] T 86fa4255c6554ba28e141de3cb328976 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Initialized, Role: FOLLOWER
I20240502 21:29:26.378994 7638 tablet_copy_service.cc:342] P 0ddd486109f5434cace13d60d633ac04: Request end of tablet copy session d0456c5e55a542128688904e003d403c-6065bdba6e2840b8b368d8f92d1fa260 received from {username='slave'} at 127.29.82.70:37961
I20240502 21:29:26.379191 7638 tablet_copy_service.cc:433] P 0ddd486109f5434cace13d60d633ac04: ending tablet copy session d0456c5e55a542128688904e003d403c-6065bdba6e2840b8b368d8f92d1fa260 on tablet 6065bdba6e2840b8b368d8f92d1fa260 with peer d0456c5e55a542128688904e003d403c
W20240502 21:29:26.382100 24439 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 21:29:26.362713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.82.67:34537 (local address 127.29.82.72:50801)
0502 21:29:26.381297 (+ 18584us) negotiation.cc:107] Waiting for socket to connect
0502 21:29:26.381420 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":18433,"thread_start_us":17071,"threads_started":1}
W20240502 21:29:26.382329 7552 consensus_peers.cc:485] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.382462 7552 consensus_peers.cc:485] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.382860 4773 connection.cc:535] client connection to 127.29.82.67:34537 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:29:26.382565 7552 consensus_peers.cc:485] T 2707427ddc3d49b89c000e92e3b663d9 P 0ddd486109f5434cace13d60d633ac04 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.383103 7552 consensus_peers.cc:485] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.383105 4773 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:29:26.383200 7552 consensus_peers.cc:485] T a8a3d57ff2184ab3bc09140afbffe0b8 P 0ddd486109f5434cace13d60d633ac04 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.383273 7552 consensus_peers.cc:485] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.398981 24440 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 21:29:26.363796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.82.67:34537 (local address 127.29.82.66:40801)
0502 21:29:26.375766 (+ 11970us) negotiation.cc:107] Waiting for socket to connect
0502 21:29:26.375788 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0502 21:29:26.375921 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 21:29:26.398291 (+ 22370us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":11852,"thread_start_us":11988,"threads_started":1}
W20240502 21:29:26.399372 4511 consensus_peers.cc:485] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 7a736a8ebb164ce0a051c0fccd76c3b8 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.399534 4511 consensus_peers.cc:485] T 91044c2ac27d42f8a804bc5eadb13d8f P 7a736a8ebb164ce0a051c0fccd76c3b8 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.399670 4511 consensus_peers.cc:485] T 016ad8b6874c4b629882f1243ea1fba2 P 7a736a8ebb164ce0a051c0fccd76c3b8 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:26.399471 24372 consensus_queue.cc:260] T 86fa4255c6554ba28e141de3cb328976 P 39981a8dfe94462b9a73ba489af4c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
W20240502 21:29:26.399753 4511 consensus_peers.cc:485] T e9f6f71aaffa41e281e67fec134a4827 P 7a736a8ebb164ce0a051c0fccd76c3b8 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.399839 4511 consensus_peers.cc:485] T e528100a026a4b429ee146912d08a20e P 7a736a8ebb164ce0a051c0fccd76c3b8 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.399945 4511 consensus_peers.cc:485] T fed543e42f844227a14a8d812ce158ba P 7a736a8ebb164ce0a051c0fccd76c3b8 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:26.400439 24372 ts_tablet_manager.cc:1398] T 86fa4255c6554ba28e141de3cb328976 P 39981a8dfe94462b9a73ba489af4c714: Time spent starting tablet: real 0.027s user 0.000s sys 0.002s
I20240502 21:29:26.402580 4859 tablet_copy_service.cc:342] P 73a1f2ca1f544de9b0e3b37015f85328: Request end of tablet copy session 39981a8dfe94462b9a73ba489af4c714-86fa4255c6554ba28e141de3cb328976 received from {username='slave'} at 127.29.82.69:57807
I20240502 21:29:26.402719 4859 tablet_copy_service.cc:433] P 73a1f2ca1f544de9b0e3b37015f85328: ending tablet copy session 39981a8dfe94462b9a73ba489af4c714-86fa4255c6554ba28e141de3cb328976 on tablet 86fa4255c6554ba28e141de3cb328976 with peer 39981a8dfe94462b9a73ba489af4c714
W20240502 21:29:26.407150 24443 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 21:29:26.379639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.82.67:34537 (local address 127.29.82.71:58775)
0502 21:29:26.406409 (+ 26770us) negotiation.cc:107] Waiting for socket to connect
0502 21:29:26.406547 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":26643,"threads_started":1}
W20240502 21:29:26.413784 7420 consensus_peers.cc:485] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.413949 7420 consensus_peers.cc:485] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.414069 7420 consensus_peers.cc:485] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.419867 4773 consensus_peers.cc:485] T 9a3c115189e8442ba7c3fa7f65089d11 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.420081 4773 consensus_peers.cc:485] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.420189 4773 consensus_peers.cc:485] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.420295 4773 consensus_peers.cc:485] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.420384 4773 consensus_peers.cc:485] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.420464 4773 consensus_peers.cc:485] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.420615 4773 consensus_peers.cc:485] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.420708 4773 consensus_peers.cc:485] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.422317 24436 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 21:29:26.356001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.82.67:34537 (local address 127.29.82.69:58445)
0502 21:29:26.421522 (+ 65521us) negotiation.cc:107] Waiting for socket to connect
0502 21:29:26.421670 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":65376,"spinlock_wait_cycles":7040,"thread_start_us":65373,"threads_started":1}
I20240502 21:29:26.423012 7212 raft_consensus.cc:1203] T dd1cf8e447e54d6e907ffa0e4681515f P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
W20240502 21:29:26.426632 24435 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 21:29:26.355353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.82.67:34537 (local address 127.29.82.70:39049)
0502 21:29:26.425944 (+ 70591us) negotiation.cc:107] Waiting for socket to connect
0502 21:29:26.426089 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":70461,"threads_started":1}
W20240502 21:29:26.428279 7285 consensus_peers.cc:485] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.428407 7285 consensus_peers.cc:462] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection reset by peer (error 104)
W20240502 21:29:26.428514 7285 consensus_peers.cc:485] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.428608 7285 consensus_peers.cc:485] T d714416dd87a49c6bd2d2b8de348c9df P d0456c5e55a542128688904e003d403c -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.431638 7151 consensus_peers.cc:485] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.431778 7151 consensus_peers.cc:485] T 56517ab7e5e447fea86868dd0a5b3c25 P 39981a8dfe94462b9a73ba489af4c714 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.431871 7151 consensus_peers.cc:485] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.431957 7151 consensus_peers.cc:485] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.435211 7151 consensus_peers.cc:485] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.435730 7285 consensus_peers.cc:462] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
W20240502 21:29:26.474998 4773 consensus_peers.cc:485] T 64aeda4ec20a4b568a475cb380d2568c P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:26.527745 7262 maintenance_manager.cc:419] P 39981a8dfe94462b9a73ba489af4c714: Scheduling UndoDeltaBlockGCOp(86fa4255c6554ba28e141de3cb328976): 4586 bytes on disk
I20240502 21:29:26.528928 7152 maintenance_manager.cc:643] P 39981a8dfe94462b9a73ba489af4c714: UndoDeltaBlockGCOp(86fa4255c6554ba28e141de3cb328976) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
W20240502 21:29:26.614801 7416 connection.cc:535] client connection to 127.29.82.65:42781 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:29:26.617506 7281 connection.cc:535] client connection to 127.29.82.65:42781 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:29:26.618438 7148 connection.cc:535] client connection to 127.29.82.65:42781 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:29:26.619647 4508 connection.cc:535] client connection to 127.29.82.65:42781 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:29:26.621500 4770 connection.cc:535] client connection to 127.29.82.65:42781 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:29:26.621506 7549 connection.cc:535] client connection to 127.29.82.65:42781 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:29:26.622550 24443 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 21:29:26.616148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.82.65:42781 (local address 127.29.82.71:35111)
0502 21:29:26.616342 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0502 21:29:26.616361 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0502 21:29:26.616483 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 21:29:26.622329 (+ 5846us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":91}
W20240502 21:29:26.623049 24440 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 21:29:26.620353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.82.65:42781 (local address 127.29.82.66:46925)
0502 21:29:26.621083 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0502 21:29:26.621101 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0502 21:29:26.621203 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 21:29:26.622882 (+ 1679us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":613}
W20240502 21:29:26.623276 4508 consensus_peers.cc:485] T 4c06f3bc97254bff8181d3f6f02470b4 P 7a736a8ebb164ce0a051c0fccd76c3b8 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.623370 24436 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 21:29:26.619950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.82.65:42781 (local address 127.29.82.69:52795)
0502 21:29:26.620331 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0502 21:29:26.620349 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0502 21:29:26.620467 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 21:29:26.623187 (+ 2720us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":265}
W20240502 21:29:26.623391 4508 consensus_peers.cc:485] T 91044c2ac27d42f8a804bc5eadb13d8f P 7a736a8ebb164ce0a051c0fccd76c3b8 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.623497 4508 consensus_peers.cc:485] T ebcf1894471f4fb7b951a730c2a781dd P 7a736a8ebb164ce0a051c0fccd76c3b8 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.623620 4508 consensus_peers.cc:485] T e528100a026a4b429ee146912d08a20e P 7a736a8ebb164ce0a051c0fccd76c3b8 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.623708 4508 consensus_peers.cc:485] T 171a71f8620c44909b78cb7b35cfed5e P 7a736a8ebb164ce0a051c0fccd76c3b8 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.623733 24435 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 21:29:26.618609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.82.65:42781 (local address 127.29.82.70:51697)
0502 21:29:26.618890 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0502 21:29:26.618909 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0502 21:29:26.619054 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 21:29:26.623415 (+ 4361us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":173}
W20240502 21:29:26.623795 4508 consensus_peers.cc:485] T e5ab4fa95c5a46e9a3377a3c96b9e740 P 7a736a8ebb164ce0a051c0fccd76c3b8 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.624133 7281 consensus_peers.cc:485] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.624267 7281 consensus_peers.cc:485] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.624650 7416 consensus_peers.cc:485] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.624776 7416 consensus_peers.cc:485] T 39260f3652b9473687ce1abb9c39223e P 31a72286397c4edc9a63eca3412be168 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.624879 7416 consensus_peers.cc:485] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.624994 7416 consensus_peers.cc:485] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.625398 4770 consensus_peers.cc:485] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.625548 4770 consensus_peers.cc:485] T 845233761c2e45b0a23e25107f5461f1 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.625599 7148 consensus_peers.cc:462] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:29:26.625658 4770 consensus_peers.cc:485] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.625726 7148 consensus_peers.cc:485] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.625749 4770 consensus_peers.cc:485] T b5099940a48f4466aa7c713bb88d108e P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.625828 7148 consensus_peers.cc:485] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.625929 7148 consensus_peers.cc:485] T 9e6feb60ae03404e85ef76499de190c3 P 39981a8dfe94462b9a73ba489af4c714 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.626016 7148 consensus_peers.cc:485] T 612a0397b1754fda81c4a0019067a0d3 P 39981a8dfe94462b9a73ba489af4c714 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.626507 7549 consensus_peers.cc:485] T beaa535c1dbe48b48613582d65e751ae P 0ddd486109f5434cace13d60d633ac04 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.626646 7549 consensus_peers.cc:485] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.626744 7549 consensus_peers.cc:485] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.626839 7549 consensus_peers.cc:485] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.626961 7549 consensus_peers.cc:485] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.639494 7549 consensus_peers.cc:485] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.667102 7281 consensus_peers.cc:485] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.673496 4770 consensus_peers.cc:485] T 0c56940914cb4161bd258bf881f06c07 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.680465 7148 consensus_peers.cc:462] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:26.688498 7347 raft_consensus.cc:1203] T 6065bdba6e2840b8b368d8f92d1fa260 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240502 21:29:26.748574 4770 consensus_peers.cc:485] T 9a3c115189e8442ba7c3fa7f65089d11 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.787533 7148 consensus_peers.cc:485] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:26.855111 7285 consensus_peers.cc:462] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:26.857583 7212 raft_consensus.cc:1203] T 86fa4255c6554ba28e141de3cb328976 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 21:29:26.875965 24305 raft_consensus.cc:482] T 5f614ef09d764740926c10618370084f P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:26.876224 24305 raft_consensus.cc:504] T 5f614ef09d764740926c10618370084f P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:26.877102 24305 leader_election.cc:288] T 5f614ef09d764740926c10618370084f P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:29:26.878366 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f614ef09d764740926c10618370084f" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
W20240502 21:29:26.879175 7151 leader_election.cc:334] T 5f614ef09d764740926c10618370084f P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:26.879446 7148 leader_election.cc:302] T 5f614ef09d764740926c10618370084f P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714; no voters: 398c3c8f3195492c900439ad5e1931bb, 73a1f2ca1f544de9b0e3b37015f85328
I20240502 21:29:26.879772 24305 raft_consensus.cc:2726] T 5f614ef09d764740926c10618370084f P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:26.888447 24032 raft_consensus.cc:482] T 2285212d78b0460ca329949eb3140e69 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:26.888659 24032 raft_consensus.cc:504] T 2285212d78b0460ca329949eb3140e69 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:26.889712 24032 leader_election.cc:288] T 2285212d78b0460ca329949eb3140e69 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:29:26.902869 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2285212d78b0460ca329949eb3140e69" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
W20240502 21:29:26.905285 7420 leader_election.cc:334] T 2285212d78b0460ca329949eb3140e69 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:26.905452 7420 leader_election.cc:302] T 2285212d78b0460ca329949eb3140e69 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 31a72286397c4edc9a63eca3412be168; no voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:29:26.905795 24032 raft_consensus.cc:2726] T 2285212d78b0460ca329949eb3140e69 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:26.906383 7597 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:26.955899 24355 raft_consensus.cc:482] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:26.956140 24355 raft_consensus.cc:504] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:26.956847 24355 leader_election.cc:288] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:29:26.956943 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2285212d78b0460ca329949eb3140e69" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
W20240502 21:29:26.962277 7549 leader_election.cc:334] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
W20240502 21:29:26.962858 7552 leader_election.cc:334] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:26.962997 7552 leader_election.cc:302] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04; no voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:26.963688 24355 raft_consensus.cc:2726] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 21:29:26.965507 7194 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 21:29:26.983533 24401 raft_consensus.cc:482] T 37c2c46368144212a0041e7ff4434236 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:26.983870 24031 raft_consensus.cc:482] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:26.983829 24401 raft_consensus.cc:504] T 37c2c46368144212a0041e7ff4434236 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:29:26.984033 24031 raft_consensus.cc:504] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:29:26.984699 24031 leader_election.cc:288] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:29:26.984792 24401 leader_election.cc:288] T 37c2c46368144212a0041e7ff4434236 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:29:26.987102 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3394f5f087304842ada4212f2e524356" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:29:26.987345 24355 raft_consensus.cc:482] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:26.987535 24355 raft_consensus.cc:504] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:26.988260 24355 leader_election.cc:288] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:29:26.990206 24150 raft_consensus.cc:482] T 0f822150741f4ec1a3a9472572160359 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:26.990399 24150 raft_consensus.cc:504] T 0f822150741f4ec1a3a9472572160359 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
W20240502 21:29:26.990572 7552 leader_election.cc:334] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:26.995234 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "546a832255ad42bebdf8eaa455ba1c3b" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:29:27.000002 24150 leader_election.cc:288] T 0f822150741f4ec1a3a9472572160359 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:29:27.001217 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f822150741f4ec1a3a9472572160359" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:29:27.003573 7552 leader_election.cc:302] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04; no voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:29:27.004112 24130 raft_consensus.cc:2726] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:29:27.008633 7285 leader_election.cc:334] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:27.008862 7285 leader_election.cc:302] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0456c5e55a542128688904e003d403c; no voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:29:27.009258 24401 raft_consensus.cc:2726] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:29:27.009459 7285 leader_election.cc:334] T 37c2c46368144212a0041e7ff4434236 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
W20240502 21:29:27.015750 7151 leader_election.cc:334] T 0f822150741f4ec1a3a9472572160359 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:27.015941 7151 leader_election.cc:302] T 0f822150741f4ec1a3a9472572160359 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714; no voters: 31a72286397c4edc9a63eca3412be168, 398c3c8f3195492c900439ad5e1931bb
I20240502 21:29:27.016319 24150 raft_consensus.cc:2726] T 0f822150741f4ec1a3a9472572160359 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:27.019915 24477 raft_consensus.cc:482] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:27.020105 24477 raft_consensus.cc:504] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:27.020978 24477 leader_election.cc:288] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:29:27.022996 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "546a832255ad42bebdf8eaa455ba1c3b" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:29:27.023224 7462 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:29:27.023236 7618 raft_consensus.cc:2454] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:29:27.023697 4509 leader_election.cc:302] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
W20240502 21:29:27.027820 4511 leader_election.cc:334] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:27.028391 24278 raft_consensus.cc:2781] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:29:27.028518 24278 raft_consensus.cc:482] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:27.028645 24278 raft_consensus.cc:3037] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:29:27.032825 7281 leader_election.cc:334] T 37c2c46368144212a0041e7ff4434236 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.032964 7281 leader_election.cc:302] T 37c2c46368144212a0041e7ff4434236 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0456c5e55a542128688904e003d403c; no voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:27.033231 24401 raft_consensus.cc:2726] T 37c2c46368144212a0041e7ff4434236 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:27.045797 24278 raft_consensus.cc:504] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:27.046403 24278 leader_election.cc:288] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:29:27.046873 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "546a832255ad42bebdf8eaa455ba1c3b" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:29:27.047132 7618 raft_consensus.cc:3037] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:29:27.048091 4511 leader_election.cc:334] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:27.049997 7618 raft_consensus.cc:2454] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 2.
I20240502 21:29:27.050534 4509 leader_election.cc:302] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters: 398c3c8f3195492c900439ad5e1931bb
I20240502 21:29:27.050817 24477 raft_consensus.cc:482] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:27.050969 24477 raft_consensus.cc:504] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:29:27.051210 24278 raft_consensus.cc:2781] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:27.051390 24278 raft_consensus.cc:686] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 2 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:29:27.051717 24278 consensus_queue.cc:237] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:27.051723 24477 leader_election.cc:288] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:29:27.052910 7347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3394f5f087304842ada4212f2e524356" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:29:27.053141 7347 raft_consensus.cc:2454] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:29:27.054888 4510 leader_election.cc:302] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a736a8ebb164ce0a051c0fccd76c3b8, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:29:27.056361 24477 raft_consensus.cc:2781] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:29:27.056469 24477 raft_consensus.cc:482] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:27.056565 24477 raft_consensus.cc:3037] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:27.059378 24477 raft_consensus.cc:504] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:29:27.059993 24477 leader_election.cc:288] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:29:27.060607 7347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3394f5f087304842ada4212f2e524356" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:29:27.060809 7347 raft_consensus.cc:3037] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:27.062003 24401 raft_consensus.cc:482] T 2ae670e3d8ae49f2b8c7df65b45b179e P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:27.062212 24401 raft_consensus.cc:504] T 2ae670e3d8ae49f2b8c7df65b45b179e P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:29:27.063043 24401 leader_election.cc:288] T 2ae670e3d8ae49f2b8c7df65b45b179e P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:29:27.063680 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae670e3d8ae49f2b8c7df65b45b179e" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:29:27.066087 7347 raft_consensus.cc:2454] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 2.
I20240502 21:29:27.066488 4510 leader_election.cc:302] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a736a8ebb164ce0a051c0fccd76c3b8, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:29:27.066720 23782 raft_consensus.cc:2781] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:27.066859 23782 raft_consensus.cc:686] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 2 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:29:27.067102 23782 consensus_queue.cc:237] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
W20240502 21:29:27.072707 7148 consensus_peers.cc:462] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.073172 24515 raft_consensus.cc:482] T 3a1fadb11cd6436a94e4a67a3aad17ba P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:27.073318 24515 raft_consensus.cc:504] T 3a1fadb11cd6436a94e4a67a3aad17ba P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:27.073983 24515 leader_election.cc:288] T 3a1fadb11cd6436a94e4a67a3aad17ba P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:29:27.075217 4573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a1fadb11cd6436a94e4a67a3aad17ba" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
W20240502 21:29:27.077661 7285 leader_election.cc:334] T 2ae670e3d8ae49f2b8c7df65b45b179e P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
W20240502 21:29:27.077718 7151 leader_election.cc:334] T 3a1fadb11cd6436a94e4a67a3aad17ba P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:27.077798 7285 leader_election.cc:302] T 2ae670e3d8ae49f2b8c7df65b45b179e P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0456c5e55a542128688904e003d403c; no voters: 0ddd486109f5434cace13d60d633ac04, 398c3c8f3195492c900439ad5e1931bb
I20240502 21:29:27.077836 7151 leader_election.cc:302] T 3a1fadb11cd6436a94e4a67a3aad17ba P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714; no voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:29:27.078441 24401 raft_consensus.cc:2726] T 2ae670e3d8ae49f2b8c7df65b45b179e P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 21:29:27.078486 24515 raft_consensus.cc:2726] T 3a1fadb11cd6436a94e4a67a3aad17ba P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:29:27.082360 4511 leader_election.cc:334] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:27.083153 24355 raft_consensus.cc:482] T a4c36adfc86141fbb8724d5a6a798c26 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:27.083335 24355 raft_consensus.cc:504] T a4c36adfc86141fbb8724d5a6a798c26 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:27.084057 24355 leader_election.cc:288] T a4c36adfc86141fbb8724d5a6a798c26 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
W20240502 21:29:27.085705 7552 leader_election.cc:334] T a4c36adfc86141fbb8724d5a6a798c26 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:27.086068 7482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c36adfc86141fbb8724d5a6a798c26" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:29:27.086788 7549 leader_election.cc:302] T a4c36adfc86141fbb8724d5a6a798c26 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04; no voters: 31a72286397c4edc9a63eca3412be168, 398c3c8f3195492c900439ad5e1931bb
I20240502 21:29:27.087213 24355 raft_consensus.cc:2726] T a4c36adfc86141fbb8724d5a6a798c26 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:29:27.088783 4511 leader_election.cc:334] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:27.093278 24520 raft_consensus.cc:482] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:27.093453 24520 raft_consensus.cc:504] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:27.094130 24520 leader_election.cc:288] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:29:27.094798 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ce97f9f9ac4398a16958931ef7262f" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:29:27.096052 24032 raft_consensus.cc:482] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:27.096299 24032 raft_consensus.cc:504] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:29:27.097160 24032 leader_election.cc:288] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
W20240502 21:29:27.100494 4511 leader_election.cc:334] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:27.100644 4511 leader_election.cc:302] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a736a8ebb164ce0a051c0fccd76c3b8; no voters: 0ddd486109f5434cace13d60d633ac04, 398c3c8f3195492c900439ad5e1931bb
I20240502 21:29:27.101146 23782 raft_consensus.cc:2726] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:29:27.104501 7416 leader_election.cc:334] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
W20240502 21:29:27.106469 7420 leader_election.cc:334] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:27.106639 7420 leader_election.cc:302] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a72286397c4edc9a63eca3412be168; no voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:27.108055 24032 raft_consensus.cc:2726] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:27.129334 4299 catalog_manager.cc:5484] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 1 to 2, leader changed from 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 2 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:29:27.130353 4817 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 21:29:27.140223 24278 raft_consensus.cc:482] T 2285212d78b0460ca329949eb3140e69 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:27.140424 24278 raft_consensus.cc:504] T 2285212d78b0460ca329949eb3140e69 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:27.140652 4555 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 21:29:27.142916 24278 leader_election.cc:288] T 2285212d78b0460ca329949eb3140e69 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:29:27.146174 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2285212d78b0460ca329949eb3140e69" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
W20240502 21:29:27.146906 4511 leader_election.cc:334] T 2285212d78b0460ca329949eb3140e69 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:27.147050 4511 leader_election.cc:302] T 2285212d78b0460ca329949eb3140e69 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7a736a8ebb164ce0a051c0fccd76c3b8; no voters: 0ddd486109f5434cace13d60d633ac04, 398c3c8f3195492c900439ad5e1931bb
I20240502 21:29:27.147110 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2285212d78b0460ca329949eb3140e69" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:29:27.147440 7484 raft_consensus.cc:2454] T 2285212d78b0460ca329949eb3140e69 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:29:27.147461 24278 raft_consensus.cc:2726] T 2285212d78b0460ca329949eb3140e69 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:27.159693 24355 raft_consensus.cc:482] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:27.159896 24355 raft_consensus.cc:504] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:27.160681 24355 leader_election.cc:288] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:29:27.161842 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2285212d78b0460ca329949eb3140e69" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:29:27.162081 4574 raft_consensus.cc:2454] T 2285212d78b0460ca329949eb3140e69 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:29:27.163391 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2285212d78b0460ca329949eb3140e69" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:29:27.163652 7484 raft_consensus.cc:2454] T 2285212d78b0460ca329949eb3140e69 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
W20240502 21:29:27.164898 7552 leader_election.cc:334] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:27.165172 7549 leader_election.cc:302] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters: 398c3c8f3195492c900439ad5e1931bb
I20240502 21:29:27.165616 24355 raft_consensus.cc:2781] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:29:27.165714 24355 raft_consensus.cc:482] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:27.165817 24355 raft_consensus.cc:3037] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:27.177470 24355 raft_consensus.cc:504] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
W20240502 21:29:27.180512 7552 leader_election.cc:334] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:27.181874 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2285212d78b0460ca329949eb3140e69" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:29:27.182147 7484 raft_consensus.cc:3037] T 2285212d78b0460ca329949eb3140e69 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:27.185086 7484 raft_consensus.cc:2454] T 2285212d78b0460ca329949eb3140e69 P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 2.
I20240502 21:29:27.187041 7328 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:29:27.188608 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2285212d78b0460ca329949eb3140e69" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:29:27.188905 4574 raft_consensus.cc:3037] T 2285212d78b0460ca329949eb3140e69 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:27.192903 4574 raft_consensus.cc:2454] T 2285212d78b0460ca329949eb3140e69 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 2.
I20240502 21:29:27.193363 7552 leader_election.cc:302] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters: 398c3c8f3195492c900439ad5e1931bb
I20240502 21:29:27.195712 24130 raft_consensus.cc:2781] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:27.197293 24130 raft_consensus.cc:686] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04 [term 2 LEADER]: Becoming Leader. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Running, Role: LEADER
I20240502 21:29:27.197644 24130 consensus_queue.cc:237] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:27.216675 24355 leader_election.cc:288] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:29:27.233835 4292 catalog_manager.cc:5484] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: term changed from 1 to 2, leader changed from 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) to 0ddd486109f5434cace13d60d633ac04 (127.29.82.72). New cstate: current_term: 2 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:27.238935 24027 raft_consensus.cc:482] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:27.239137 24027 raft_consensus.cc:504] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:27.239872 24027 leader_election.cc:288] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:29:27.240630 7212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f614ef09d764740926c10618370084f" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:29:27.240882 7212 raft_consensus.cc:2454] T 5f614ef09d764740926c10618370084f P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:29:27.241417 4772 leader_election.cc:302] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:29:27.241813 24027 raft_consensus.cc:2781] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:29:27.241925 24027 raft_consensus.cc:482] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:27.242036 24027 raft_consensus.cc:3037] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:27.242890 23735 raft_consensus.cc:482] T 0914e3ce5a2d43788ba9d369227e4f66 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.243048 23735 raft_consensus.cc:504] T 0914e3ce5a2d43788ba9d369227e4f66 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:27.243816 23735 leader_election.cc:288] T 0914e3ce5a2d43788ba9d369227e4f66 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:29:27.244916 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0914e3ce5a2d43788ba9d369227e4f66" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:29:27.245288 24027 raft_consensus.cc:504] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
W20240502 21:29:27.247596 4770 leader_election.cc:334] T 0914e3ce5a2d43788ba9d369227e4f66 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.247738 4770 leader_election.cc:302] T 0914e3ce5a2d43788ba9d369227e4f66 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328; no voters: 0ddd486109f5434cace13d60d633ac04, 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:27.247937 7212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f614ef09d764740926c10618370084f" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:29:27.248147 7212 raft_consensus.cc:3037] T 5f614ef09d764740926c10618370084f P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:29:27.249287 4773 leader_election.cc:334] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
W20240502 21:29:27.249533 4773 leader_election.cc:334] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:27.250118 24525 raft_consensus.cc:482] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:27.250298 24525 raft_consensus.cc:504] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:29:27.250886 24525 leader_election.cc:288] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:29:27.251148 7212 raft_consensus.cc:2454] T 5f614ef09d764740926c10618370084f P 39981a8dfe94462b9a73ba489af4c714 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 2.
I20240502 21:29:27.252993 7347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae670e3d8ae49f2b8c7df65b45b179e" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:29:27.253216 7347 raft_consensus.cc:2454] T 2ae670e3d8ae49f2b8c7df65b45b179e P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 2.
I20240502 21:29:27.253580 7551 leader_election.cc:302] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, d0456c5e55a542128688904e003d403c; no voters:
W20240502 21:29:27.253803 7552 leader_election.cc:334] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:27.254220 4772 leader_election.cc:302] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 73a1f2ca1f544de9b0e3b37015f85328; no voters: 398c3c8f3195492c900439ad5e1931bb
I20240502 21:29:27.254578 23735 raft_consensus.cc:2726] T 0914e3ce5a2d43788ba9d369227e4f66 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:27.254760 23735 raft_consensus.cc:2781] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:27.254904 23735 raft_consensus.cc:686] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328 [term 2 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:29:27.255187 23735 consensus_queue.cc:237] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:27.256006 24530 raft_consensus.cc:2781] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 21:29:27.256132 24530 raft_consensus.cc:482] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:27.256256 24530 raft_consensus.cc:3037] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:27.259195 24530 raft_consensus.cc:504] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:29:27.260244 24530 leader_election.cc:288] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 3 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:29:27.260217 7347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae670e3d8ae49f2b8c7df65b45b179e" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:29:27.260455 7347 raft_consensus.cc:3037] T 2ae670e3d8ae49f2b8c7df65b45b179e P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:27.261687 24027 leader_election.cc:288] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 2 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
W20240502 21:29:27.265667 7552 leader_election.cc:334] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:27.265494 4299 catalog_manager.cc:5484] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 1 to 2, leader changed from 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 2 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } }
I20240502 21:29:27.266307 7347 raft_consensus.cc:2454] T 2ae670e3d8ae49f2b8c7df65b45b179e P d0456c5e55a542128688904e003d403c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 3.
I20240502 21:29:27.266788 7551 leader_election.cc:302] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, d0456c5e55a542128688904e003d403c; no voters: 398c3c8f3195492c900439ad5e1931bb
I20240502 21:29:27.267236 24530 raft_consensus.cc:2781] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:27.267552 24530 raft_consensus.cc:686] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 [term 3 LEADER]: Becoming Leader. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Running, Role: LEADER
I20240502 21:29:27.267968 24530 consensus_queue.cc:237] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:29:27.278331 24401 raft_consensus.cc:482] T 1bddb5e40d464de082c1d586618c86d7 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.278497 24401 raft_consensus.cc:504] T 1bddb5e40d464de082c1d586618c86d7 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:29:27.279192 24401 leader_election.cc:288] T 1bddb5e40d464de082c1d586618c86d7 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:29:27.283751 4299 catalog_manager.cc:5484] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 1 to 2, leader changed from 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 2 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:27.285351 24525 raft_consensus.cc:482] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.285531 24525 raft_consensus.cc:504] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:27.286360 24525 leader_election.cc:288] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:29:27.287230 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0914e3ce5a2d43788ba9d369227e4f66" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:29:27.287479 4838 raft_consensus.cc:2454] T 0914e3ce5a2d43788ba9d369227e4f66 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:29:27.288051 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bddb5e40d464de082c1d586618c86d7" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:29:27.288762 7549 leader_election.cc:302] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:29:27.289366 24527 raft_consensus.cc:2781] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:29:27.289501 24527 raft_consensus.cc:482] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.289584 24527 raft_consensus.cc:3037] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:29:27.293772 7549 leader_election.cc:334] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.296003 24527 raft_consensus.cc:504] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:27.297391 24527 leader_election.cc:288] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 election: Requested vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:29:27.298300 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0914e3ce5a2d43788ba9d369227e4f66" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:29:27.298518 4838 raft_consensus.cc:3037] T 0914e3ce5a2d43788ba9d369227e4f66 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:27.301280 4838 raft_consensus.cc:2454] T 0914e3ce5a2d43788ba9d369227e4f66 P 73a1f2ca1f544de9b0e3b37015f85328 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 2.
I20240502 21:29:27.302589 7549 leader_election.cc:302] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:29:27.303155 4299 catalog_manager.cc:5484] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 reported cstate change: term changed from 2 to 3, leader changed from 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) to 0ddd486109f5434cace13d60d633ac04 (127.29.82.72). New cstate: current_term: 3 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } }
W20240502 21:29:27.303881 7281 leader_election.cc:334] T 1bddb5e40d464de082c1d586618c86d7 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.304020 7281 leader_election.cc:302] T 1bddb5e40d464de082c1d586618c86d7 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0456c5e55a542128688904e003d403c; no voters: 0ddd486109f5434cace13d60d633ac04, 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:27.304456 24401 raft_consensus.cc:2726] T 1bddb5e40d464de082c1d586618c86d7 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:29:27.306015 7549 leader_election.cc:334] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.306228 24527 raft_consensus.cc:2781] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:27.306828 24527 raft_consensus.cc:686] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04 [term 2 LEADER]: Becoming Leader. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Running, Role: LEADER
I20240502 21:29:27.307166 24527 consensus_queue.cc:237] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:27.316996 4299 catalog_manager.cc:5484] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: term changed from 1 to 2, leader changed from 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) to 0ddd486109f5434cace13d60d633ac04 (127.29.82.72). New cstate: current_term: 2 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
W20240502 21:29:27.323614 7285 consensus_peers.cc:462] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:27.327060 24504 raft_consensus.cc:482] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:27.327234 24504 raft_consensus.cc:504] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:27.328003 24504 leader_election.cc:288] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:29:27.328532 7212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f822150741f4ec1a3a9472572160359" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:29:27.328768 7212 raft_consensus.cc:2454] T 0f822150741f4ec1a3a9472572160359 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:29:27.329227 7419 leader_election.cc:302] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:29:27.329515 24504 raft_consensus.cc:2781] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:29:27.329619 24504 raft_consensus.cc:482] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:27.329725 24504 raft_consensus.cc:3037] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:29:27.330344 7420 leader_election.cc:334] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:27.334834 24504 raft_consensus.cc:504] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:27.336220 7212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f822150741f4ec1a3a9472572160359" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:29:27.336424 7212 raft_consensus.cc:3037] T 0f822150741f4ec1a3a9472572160359 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:27.336795 24527 raft_consensus.cc:482] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:27.336956 24527 raft_consensus.cc:504] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
W20240502 21:29:27.337101 7420 leader_election.cc:334] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:27.337601 24527 leader_election.cc:288] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:29:27.338223 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ce97f9f9ac4398a16958931ef7262f" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:29:27.338428 4574 raft_consensus.cc:2454] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:29:27.338774 7552 leader_election.cc:302] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:29:27.339048 24527 raft_consensus.cc:2781] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:29:27.339107 7212 raft_consensus.cc:2454] T 0f822150741f4ec1a3a9472572160359 P 39981a8dfe94462b9a73ba489af4c714 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 2.
I20240502 21:29:27.339241 24527 raft_consensus.cc:482] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:27.339385 24527 raft_consensus.cc:3037] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:29:27.339773 7552 leader_election.cc:334] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:27.339879 7419 leader_election.cc:302] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 39981a8dfe94462b9a73ba489af4c714; no voters: 398c3c8f3195492c900439ad5e1931bb
I20240502 21:29:27.340159 24032 raft_consensus.cc:2781] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:27.341027 24504 leader_election.cc:288] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:29:27.342613 24527 raft_consensus.cc:504] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:27.344372 24527 leader_election.cc:288] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
W20240502 21:29:27.345027 7552 leader_election.cc:334] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:27.345057 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ce97f9f9ac4398a16958931ef7262f" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8"
I20240502 21:29:27.345224 4574 raft_consensus.cc:3037] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:27.345530 23735 raft_consensus.cc:482] T a7a0167853fa420586455b82fb893c46 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.345683 23735 raft_consensus.cc:504] T a7a0167853fa420586455b82fb893c46 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:27.346288 23735 leader_election.cc:288] T a7a0167853fa420586455b82fb893c46 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
W20240502 21:29:27.347836 4770 leader_election.cc:334] T a7a0167853fa420586455b82fb893c46 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.348122 4574 raft_consensus.cc:2454] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 2.
I20240502 21:29:27.348259 24538 raft_consensus.cc:482] T 128dc7838caa403282ed2868444be271 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:27.348431 24538 raft_consensus.cc:504] T 128dc7838caa403282ed2868444be271 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:27.350423 7552 leader_election.cc:302] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters: 398c3c8f3195492c900439ad5e1931bb
I20240502 21:29:27.350893 24527 raft_consensus.cc:2781] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:27.351035 24527 raft_consensus.cc:686] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04 [term 2 LEADER]: Becoming Leader. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Running, Role: LEADER
I20240502 21:29:27.351626 24527 consensus_queue.cc:237] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:27.351729 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a0167853fa420586455b82fb893c46" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:29:27.352702 24032 raft_consensus.cc:686] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 [term 2 LEADER]: Becoming Leader. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Running, Role: LEADER
W20240502 21:29:27.352895 7416 leader_election.cc:334] T 128dc7838caa403282ed2868444be271 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.352998 24032 consensus_queue.cc:237] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:27.353382 4771 leader_election.cc:302] T a7a0167853fa420586455b82fb893c46 P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328; no voters: 0ddd486109f5434cace13d60d633ac04, 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:27.353842 23735 raft_consensus.cc:2726] T a7a0167853fa420586455b82fb893c46 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:29:27.361433 7420 leader_election.cc:334] T 128dc7838caa403282ed2868444be271 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:27.361595 7420 leader_election.cc:302] T 128dc7838caa403282ed2868444be271 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a72286397c4edc9a63eca3412be168; no voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:27.361984 24504 raft_consensus.cc:2726] T 128dc7838caa403282ed2868444be271 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:27.363687 24538 leader_election.cc:288] T 128dc7838caa403282ed2868444be271 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:29:27.368263 24401 raft_consensus.cc:482] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.368438 24401 raft_consensus.cc:504] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:29:27.369454 24401 leader_election.cc:288] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
W20240502 21:29:27.371477 7281 leader_election.cc:334] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.372148 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292bfad92b14456da85bdc874fb7e06e" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:29:27.372778 7281 leader_election.cc:302] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0456c5e55a542128688904e003d403c; no voters: 71e1375b799b439fa3c83ca763c039c1, 73a1f2ca1f544de9b0e3b37015f85328
I20240502 21:29:27.373175 24401 raft_consensus.cc:2726] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:27.374082 4299 catalog_manager.cc:5484] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 reported cstate change: term changed from 1 to 2, leader changed from 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) to 31a72286397c4edc9a63eca3412be168 (127.29.82.71). New cstate: current_term: 2 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:27.376062 4292 catalog_manager.cc:5484] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04 reported cstate change: term changed from 1 to 2, leader changed from 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) to 0ddd486109f5434cace13d60d633ac04 (127.29.82.72). New cstate: current_term: 2 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:29:27.377568 23782 raft_consensus.cc:482] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:27.377712 23782 raft_consensus.cc:504] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:27.378382 23782 leader_election.cc:288] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:29:27.379194 7212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a1fadb11cd6436a94e4a67a3aad17ba" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:29:27.379441 7212 raft_consensus.cc:2454] T 3a1fadb11cd6436a94e4a67a3aad17ba P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:29:27.379878 4510 leader_election.cc:302] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:29:27.380097 23782 raft_consensus.cc:2781] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:29:27.380177 23782 raft_consensus.cc:482] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:27.380249 23782 raft_consensus.cc:3037] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:27.382110 23735 raft_consensus.cc:482] T fb2cc722452f4d9798353d1bfae4bd4f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.382257 23735 raft_consensus.cc:504] T fb2cc722452f4d9798353d1bfae4bd4f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:29:27.383199 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb2cc722452f4d9798353d1bfae4bd4f" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:29:27.383409 23735 leader_election.cc:288] T fb2cc722452f4d9798353d1bfae4bd4f P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:29:27.383807 23782 raft_consensus.cc:504] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
W20240502 21:29:27.383884 4511 leader_election.cc:334] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
W20240502 21:29:27.384850 4770 leader_election.cc:334] T fb2cc722452f4d9798353d1bfae4bd4f P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.384863 7212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a1fadb11cd6436a94e4a67a3aad17ba" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714"
I20240502 21:29:27.384956 4770 leader_election.cc:302] T fb2cc722452f4d9798353d1bfae4bd4f P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328; no voters: 31a72286397c4edc9a63eca3412be168, 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:27.385071 7212 raft_consensus.cc:3037] T 3a1fadb11cd6436a94e4a67a3aad17ba P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:27.385375 23735 raft_consensus.cc:2726] T fb2cc722452f4d9798353d1bfae4bd4f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:27.387676 7212 raft_consensus.cc:2454] T 3a1fadb11cd6436a94e4a67a3aad17ba P 39981a8dfe94462b9a73ba489af4c714 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 2.
I20240502 21:29:27.388120 4510 leader_election.cc:302] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:29:27.388368 24278 raft_consensus.cc:2781] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:27.388513 24278 raft_consensus.cc:686] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 2 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:29:27.388777 24278 consensus_queue.cc:237] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
W20240502 21:29:27.389910 4511 leader_election.cc:334] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:27.388280 23782 leader_election.cc:288] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:29:27.393288 24032 raft_consensus.cc:482] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.393453 24032 raft_consensus.cc:504] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:27.394083 24032 leader_election.cc:288] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:29:27.394520 4299 catalog_manager.cc:5484] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 1 to 2, leader changed from 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 2 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
W20240502 21:29:27.395601 7416 leader_election.cc:334] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.395637 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eb73e73b64b4b0f9e5116f258f2e147" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:29:27.396301 7420 leader_election.cc:302] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a72286397c4edc9a63eca3412be168; no voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:29:27.396589 24032 raft_consensus.cc:2726] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:27.401871 24032 raft_consensus.cc:482] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:27.402040 24032 raft_consensus.cc:504] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:27.403028 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c36adfc86141fbb8724d5a6a798c26" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:29:27.403293 7618 raft_consensus.cc:2454] T a4c36adfc86141fbb8724d5a6a798c26 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:29:27.403786 7418 leader_election.cc:302] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168; no voters:
I20240502 21:29:27.404072 24538 raft_consensus.cc:2781] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:29:27.404165 24538 raft_consensus.cc:482] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:27.404265 24538 raft_consensus.cc:3037] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:29:27.405777 7420 leader_election.cc:334] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:27.406256 24538 raft_consensus.cc:504] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:27.406749 24538 leader_election.cc:288] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 election: Requested vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:29:27.407193 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c36adfc86141fbb8724d5a6a798c26" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:29:27.407389 7618 raft_consensus.cc:3037] T a4c36adfc86141fbb8724d5a6a798c26 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:29:27.408089 7420 leader_election.cc:334] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:27.409927 7618 raft_consensus.cc:2454] T a4c36adfc86141fbb8724d5a6a798c26 P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 2.
I20240502 21:29:27.410547 7418 leader_election.cc:302] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168; no voters: 398c3c8f3195492c900439ad5e1931bb
I20240502 21:29:27.411407 24537 raft_consensus.cc:2781] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:27.411564 24537 raft_consensus.cc:686] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168 [term 2 LEADER]: Becoming Leader. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Running, Role: LEADER
I20240502 21:29:27.411892 24537 consensus_queue.cc:237] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:27.412199 24032 leader_election.cc:288] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:29:27.417752 4299 catalog_manager.cc:5484] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168 reported cstate change: term changed from 1 to 2, leader changed from 398c3c8f3195492c900439ad5e1931bb (127.29.82.67) to 31a72286397c4edc9a63eca3412be168 (127.29.82.71). New cstate: current_term: 2 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:27.427098 24527 raft_consensus.cc:482] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.427325 24527 raft_consensus.cc:504] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:27.428100 24527 leader_election.cc:288] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:29:27.428864 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a0167853fa420586455b82fb893c46" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:29:27.429085 4838 raft_consensus.cc:2454] T a7a0167853fa420586455b82fb893c46 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
W20240502 21:29:27.429852 7549 leader_election.cc:334] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.430191 7549 leader_election.cc:302] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 73a1f2ca1f544de9b0e3b37015f85328; no voters: 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:27.430480 24530 raft_consensus.cc:2781] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:29:27.430588 24530 raft_consensus.cc:482] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.430704 24530 raft_consensus.cc:3037] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:27.433050 24530 raft_consensus.cc:504] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:27.433676 24530 leader_election.cc:288] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 election: Requested vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:29:27.434366 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a0167853fa420586455b82fb893c46" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:29:27.434587 4838 raft_consensus.cc:3037] T a7a0167853fa420586455b82fb893c46 P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:27.436750 4838 raft_consensus.cc:2454] T a7a0167853fa420586455b82fb893c46 P 73a1f2ca1f544de9b0e3b37015f85328 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 2.
I20240502 21:29:27.437186 7549 leader_election.cc:302] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 73a1f2ca1f544de9b0e3b37015f85328; no voters:
I20240502 21:29:27.437438 24530 raft_consensus.cc:2781] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Leader election won for term 2
W20240502 21:29:27.437446 7549 leader_election.cc:334] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.437587 24530 raft_consensus.cc:686] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04 [term 2 LEADER]: Becoming Leader. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Running, Role: LEADER
I20240502 21:29:27.437824 24530 consensus_queue.cc:237] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:27.442127 4299 catalog_manager.cc:5484] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: term changed from 1 to 2, leader changed from 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) to 0ddd486109f5434cace13d60d633ac04 (127.29.82.72). New cstate: current_term: 2 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
I20240502 21:29:27.444059 24401 raft_consensus.cc:482] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:27.444253 24401 raft_consensus.cc:504] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:29:27.444929 24401 leader_election.cc:288] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
W20240502 21:29:27.446939 7281 leader_election.cc:334] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.447203 23782 raft_consensus.cc:482] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.447382 23782 raft_consensus.cc:504] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
W20240502 21:29:27.447763 7285 leader_election.cc:334] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:27.447878 7285 leader_election.cc:302] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0456c5e55a542128688904e003d403c; no voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:27.448093 23782 leader_election.cc:288] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:29:27.448374 24401 raft_consensus.cc:2726] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:27.448957 24537 raft_consensus.cc:482] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.448999 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eb73e73b64b4b0f9e5116f258f2e147" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:29:27.449218 7484 raft_consensus.cc:2454] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:29:27.449167 24537 raft_consensus.cc:504] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:29:27.449666 4508 leader_election.cc:302] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters:
I20240502 21:29:27.449973 24537 leader_election.cc:288] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:29:27.450664 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb2cc722452f4d9798353d1bfae4bd4f" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328" is_pre_election: true
I20240502 21:29:27.450830 23782 raft_consensus.cc:2781] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:29:27.450945 23782 raft_consensus.cc:482] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.451056 23782 raft_consensus.cc:3037] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:27.451004 4838 raft_consensus.cc:2454] T fb2cc722452f4d9798353d1bfae4bd4f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
W20240502 21:29:27.451506 7416 leader_election.cc:334] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.453202 23782 raft_consensus.cc:504] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:27.453802 23782 leader_election.cc:288] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:29:27.454382 7416 leader_election.cc:302] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 73a1f2ca1f544de9b0e3b37015f85328; no voters: 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:27.454866 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eb73e73b64b4b0f9e5116f258f2e147" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:29:27.455066 7484 raft_consensus.cc:3037] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:27.455117 24401 raft_consensus.cc:482] T 6687a533112445dc86951fc778d0725b P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.455260 24401 raft_consensus.cc:504] T 6687a533112445dc86951fc778d0725b P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:29:27.455888 24401 leader_election.cc:288] T 6687a533112445dc86951fc778d0725b P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
W20240502 21:29:27.457355 7281 leader_election.cc:334] T 6687a533112445dc86951fc778d0725b P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
W20240502 21:29:27.458143 4508 leader_election.cc:334] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
W20240502 21:29:27.459760 7285 leader_election.cc:334] T 6687a533112445dc86951fc778d0725b P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:27.459878 7285 leader_election.cc:302] T 6687a533112445dc86951fc778d0725b P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0456c5e55a542128688904e003d403c; no voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:27.460260 24401 raft_consensus.cc:2726] T 6687a533112445dc86951fc778d0725b P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:27.462582 24032 raft_consensus.cc:2781] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:29:27.462739 24032 raft_consensus.cc:482] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.462864 24032 raft_consensus.cc:3037] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:29:27.463137 4508 leader_election.cc:334] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.475468 24401 raft_consensus.cc:482] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.475744 24401 raft_consensus.cc:504] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:27.476583 24401 leader_election.cc:288] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:29:27.475462 24031 raft_consensus.cc:482] T 5d710d1340514764b0c7d38e6bf19436 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.476837 24031 raft_consensus.cc:504] T 5d710d1340514764b0c7d38e6bf19436 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:29:27.477365 24031 leader_election.cc:288] T 5d710d1340514764b0c7d38e6bf19436 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:29:27.478686 7482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c243b27429043aa8642d9b2ed595ab3" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:29:27.478928 7482 raft_consensus.cc:2454] T 1c243b27429043aa8642d9b2ed595ab3 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
I20240502 21:29:27.480468 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d710d1340514764b0c7d38e6bf19436" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
W20240502 21:29:27.481240 7281 leader_election.cc:334] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
W20240502 21:29:27.481384 7281 leader_election.cc:334] T 5d710d1340514764b0c7d38e6bf19436 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.481881 7285 leader_election.cc:302] T 5d710d1340514764b0c7d38e6bf19436 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0456c5e55a542128688904e003d403c; no voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:29:27.482334 24031 raft_consensus.cc:2726] T 5d710d1340514764b0c7d38e6bf19436 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:29:27.482656 7285 leader_election.cc:334] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:27.482771 7285 leader_election.cc:302] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, d0456c5e55a542128688904e003d403c; no voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:27.483341 24031 raft_consensus.cc:2726] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:27.471561 24032 raft_consensus.cc:504] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:29:27.490331 24031 raft_consensus.cc:482] T ca332df7aaef4ddd9fd3266e8003b55f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
W20240502 21:29:27.490368 7416 leader_election.cc:334] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.490483 24031 raft_consensus.cc:504] T ca332df7aaef4ddd9fd3266e8003b55f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:27.490648 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb2cc722452f4d9798353d1bfae4bd4f" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "73a1f2ca1f544de9b0e3b37015f85328"
I20240502 21:29:27.490840 4838 raft_consensus.cc:3037] T fb2cc722452f4d9798353d1bfae4bd4f P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:27.491138 24031 leader_election.cc:288] T ca332df7aaef4ddd9fd3266e8003b55f P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:29:27.491605 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca332df7aaef4ddd9fd3266e8003b55f" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
W20240502 21:29:27.493372 7281 leader_election.cc:334] T ca332df7aaef4ddd9fd3266e8003b55f P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.493999 4838 raft_consensus.cc:2454] T fb2cc722452f4d9798353d1bfae4bd4f P 73a1f2ca1f544de9b0e3b37015f85328 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 2.
I20240502 21:29:27.494623 7416 leader_election.cc:302] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 73a1f2ca1f544de9b0e3b37015f85328; no voters: 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:27.495014 24537 raft_consensus.cc:2781] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:27.495167 24537 raft_consensus.cc:686] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 [term 2 LEADER]: Becoming Leader. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Running, Role: LEADER
I20240502 21:29:27.495465 24537 consensus_queue.cc:237] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:29:27.496245 24530 raft_consensus.cc:482] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.496416 24530 raft_consensus.cc:504] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:27.497045 24530 leader_election.cc:288] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:29:27.489991 24032 leader_election.cc:288] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:29:27.500144 7347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca332df7aaef4ddd9fd3266e8003b55f" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:29:27.500363 7347 raft_consensus.cc:2454] T ca332df7aaef4ddd9fd3266e8003b55f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:29:27.500730 7551 leader_election.cc:302] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:29:27.501036 24530 raft_consensus.cc:2781] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:29:27.501147 24530 raft_consensus.cc:482] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.501233 24530 raft_consensus.cc:3037] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:27.504674 24530 raft_consensus.cc:504] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:27.505939 24530 leader_election.cc:288] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:29:27.506567 7347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca332df7aaef4ddd9fd3266e8003b55f" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:29:27.506829 7347 raft_consensus.cc:3037] T ca332df7aaef4ddd9fd3266e8003b55f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:27.513608 7347 raft_consensus.cc:2454] T ca332df7aaef4ddd9fd3266e8003b55f P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 2.
I20240502 21:29:27.514192 7551 leader_election.cc:302] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:29:27.514549 24530 raft_consensus.cc:2781] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:27.514755 24530 raft_consensus.cc:686] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04 [term 2 LEADER]: Becoming Leader. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Running, Role: LEADER
I20240502 21:29:27.515273 24530 consensus_queue.cc:237] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:27.518072 4299 catalog_manager.cc:5484] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 reported cstate change: term changed from 1 to 2, leader changed from 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) to 31a72286397c4edc9a63eca3412be168 (127.29.82.71). New cstate: current_term: 2 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } }
W20240502 21:29:27.522275 7549 leader_election.cc:334] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
W20240502 21:29:27.522531 7549 leader_election.cc:334] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.522987 24031 raft_consensus.cc:482] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.523128 24031 raft_consensus.cc:504] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:29:27.523738 24031 leader_election.cc:288] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:29:27.524057 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bd28b4c6a54d98a18351b60f299a3a" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" is_pre_election: true
I20240502 21:29:27.529932 7281 leader_election.cc:302] T ca332df7aaef4ddd9fd3266e8003b55f P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0456c5e55a542128688904e003d403c; no voters: 0ddd486109f5434cace13d60d633ac04, 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:27.530237 24031 raft_consensus.cc:2726] T ca332df7aaef4ddd9fd3266e8003b55f P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:29:27.532080 4511 consensus_peers.cc:485] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:27.532300 4299 catalog_manager.cc:5484] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04 reported cstate change: term changed from 1 to 2, leader changed from 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) to 0ddd486109f5434cace13d60d633ac04 (127.29.82.72). New cstate: current_term: 2 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
W20240502 21:29:27.536448 7281 leader_election.cc:334] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.536542 7281 leader_election.cc:302] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0456c5e55a542128688904e003d403c; no voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:29:27.536741 24401 raft_consensus.cc:2726] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:27.544888 7484 raft_consensus.cc:2454] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 2.
I20240502 21:29:27.545403 4508 leader_election.cc:302] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 7a736a8ebb164ce0a051c0fccd76c3b8; no voters: 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:27.545691 23782 raft_consensus.cc:2781] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:27.545845 23782 raft_consensus.cc:686] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 2 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:29:27.546124 23782 consensus_queue.cc:237] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:27.550736 4299 catalog_manager.cc:5484] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 1 to 2, leader changed from 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 2 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:27.589807 23782 raft_consensus.cc:482] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.589993 23782 raft_consensus.cc:504] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:29:27.590646 23782 leader_election.cc:288] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:29:27.591194 7347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bd28b4c6a54d98a18351b60f299a3a" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:29:27.591439 7347 raft_consensus.cc:2454] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:29:27.591997 4510 leader_election.cc:302] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a736a8ebb164ce0a051c0fccd76c3b8, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:29:27.592301 24278 raft_consensus.cc:2781] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:29:27.592406 24278 raft_consensus.cc:482] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.592415 7618 raft_consensus.cc:1261] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
W20240502 21:29:27.592434 4508 leader_election.cc:334] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.592517 24278 raft_consensus.cc:3037] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:27.593039 23782 consensus_queue.cc:1035] T 546a832255ad42bebdf8eaa455ba1c3b P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:27.597714 24278 raft_consensus.cc:504] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:29:27.598352 24278 leader_election.cc:288] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:29:27.598929 7347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bd28b4c6a54d98a18351b60f299a3a" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:29:27.599144 7347 raft_consensus.cc:3037] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:27.601949 7347 raft_consensus.cc:2454] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 2.
I20240502 21:29:27.602330 4510 leader_election.cc:302] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a736a8ebb164ce0a051c0fccd76c3b8, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:29:27.602716 24278 raft_consensus.cc:2781] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 2 FOLLOWER]: Leader election won for term 2
W20240502 21:29:27.604872 4508 leader_election.cc:334] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.605787 24278 raft_consensus.cc:686] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 2 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:29:27.606061 24278 consensus_queue.cc:237] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:29:27.610237 24278 raft_consensus.cc:482] T 5d710d1340514764b0c7d38e6bf19436 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.610395 24278 raft_consensus.cc:504] T 5d710d1340514764b0c7d38e6bf19436 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:29:27.611004 24278 leader_election.cc:288] T 5d710d1340514764b0c7d38e6bf19436 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:29:27.611069 4299 catalog_manager.cc:5484] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 1 to 2, leader changed from 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 2 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:29:27.611964 7347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d710d1340514764b0c7d38e6bf19436" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:29:27.612167 7347 raft_consensus.cc:2454] T 5d710d1340514764b0c7d38e6bf19436 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 1.
I20240502 21:29:27.612752 4510 leader_election.cc:302] T 5d710d1340514764b0c7d38e6bf19436 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a736a8ebb164ce0a051c0fccd76c3b8, d0456c5e55a542128688904e003d403c; no voters:
W20240502 21:29:27.613299 4508 leader_election.cc:334] T 5d710d1340514764b0c7d38e6bf19436 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.614588 24278 raft_consensus.cc:2781] T 5d710d1340514764b0c7d38e6bf19436 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:29:27.614706 24278 raft_consensus.cc:482] T 5d710d1340514764b0c7d38e6bf19436 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.614799 24278 raft_consensus.cc:3037] T 5d710d1340514764b0c7d38e6bf19436 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:27.616129 7484 raft_consensus.cc:1261] T 2285212d78b0460ca329949eb3140e69 P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:29:27.616828 24527 consensus_queue.cc:1035] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:27.619880 24278 raft_consensus.cc:504] T 5d710d1340514764b0c7d38e6bf19436 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:29:27.620762 7347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d710d1340514764b0c7d38e6bf19436" candidate_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:29:27.620945 7347 raft_consensus.cc:3037] T 5d710d1340514764b0c7d38e6bf19436 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:27.623565 7347 raft_consensus.cc:2454] T 5d710d1340514764b0c7d38e6bf19436 P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a736a8ebb164ce0a051c0fccd76c3b8 in term 2.
I20240502 21:29:27.624085 4510 leader_election.cc:302] T 5d710d1340514764b0c7d38e6bf19436 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a736a8ebb164ce0a051c0fccd76c3b8, d0456c5e55a542128688904e003d403c; no voters:
I20240502 21:29:27.624397 23782 raft_consensus.cc:2781] T 5d710d1340514764b0c7d38e6bf19436 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:27.624555 23782 raft_consensus.cc:686] T 5d710d1340514764b0c7d38e6bf19436 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 2 LEADER]: Becoming Leader. State: Replica: 7a736a8ebb164ce0a051c0fccd76c3b8, State: Running, Role: LEADER
I20240502 21:29:27.624828 23782 consensus_queue.cc:237] T 5d710d1340514764b0c7d38e6bf19436 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:29:27.627980 24278 leader_election.cc:288] T 5d710d1340514764b0c7d38e6bf19436 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 election: Requested vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
W20240502 21:29:27.654105 7148 consensus_peers.cc:462] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.654646 7212 raft_consensus.cc:1261] T 5f614ef09d764740926c10618370084f P 39981a8dfe94462b9a73ba489af4c714 [term 2 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:29:27.655234 23735 consensus_queue.cc:1035] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:27.659282 4299 catalog_manager.cc:5484] T 5d710d1340514764b0c7d38e6bf19436 P 7a736a8ebb164ce0a051c0fccd76c3b8 reported cstate change: term changed from 1 to 2, leader changed from 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) to 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66). New cstate: current_term: 2 leader_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } }
I20240502 21:29:27.673621 24027 raft_consensus.cc:482] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.673803 24027 raft_consensus.cc:504] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:29:27.674652 24027 leader_election.cc:288] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
W20240502 21:29:27.676263 4770 leader_election.cc:334] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.676950 7347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292bfad92b14456da85bdc874fb7e06e" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:29:27.677165 7347 raft_consensus.cc:2454] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 1.
I20240502 21:29:27.677664 4772 leader_election.cc:302] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, d0456c5e55a542128688904e003d403c; no voters: 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:27.678427 24532 raft_consensus.cc:2781] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:29:27.678530 24532 raft_consensus.cc:482] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.678643 24532 raft_consensus.cc:3037] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:27.679385 7347 raft_consensus.cc:1261] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:29:27.679833 24537 raft_consensus.cc:482] T b589c7da65e34002b56cbf867d033c76 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.679941 23782 consensus_queue.cc:1035] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:27.680004 24537 raft_consensus.cc:504] T b589c7da65e34002b56cbf867d033c76 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:27.680699 24537 leader_election.cc:288] T b589c7da65e34002b56cbf867d033c76 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:29:27.682332 7213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b589c7da65e34002b56cbf867d033c76" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
W20240502 21:29:27.683329 7416 leader_election.cc:334] T b589c7da65e34002b56cbf867d033c76 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.683437 7416 leader_election.cc:302] T b589c7da65e34002b56cbf867d033c76 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a72286397c4edc9a63eca3412be168; no voters: 39981a8dfe94462b9a73ba489af4c714, 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:27.684127 24032 raft_consensus.cc:2726] T b589c7da65e34002b56cbf867d033c76 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:29:27.686779 4511 consensus_peers.cc:485] T 3394f5f087304842ada4212f2e524356 P 7a736a8ebb164ce0a051c0fccd76c3b8 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:27.688370 4508 leader_election.cc:334] T 5d710d1340514764b0c7d38e6bf19436 P 7a736a8ebb164ce0a051c0fccd76c3b8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.690707 24532 raft_consensus.cc:504] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
W20240502 21:29:27.692703 4770 leader_election.cc:334] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.693099 24532 leader_election.cc:288] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 2 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:29:27.693681 7347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292bfad92b14456da85bdc874fb7e06e" candidate_uuid: "73a1f2ca1f544de9b0e3b37015f85328" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:29:27.693925 7347 raft_consensus.cc:3037] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:27.697460 7347 raft_consensus.cc:2454] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a1f2ca1f544de9b0e3b37015f85328 in term 2.
I20240502 21:29:27.698192 4772 leader_election.cc:302] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73a1f2ca1f544de9b0e3b37015f85328, d0456c5e55a542128688904e003d403c; no voters: 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:27.698894 24532 raft_consensus.cc:2781] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:27.699148 24532 raft_consensus.cc:686] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328 [term 2 LEADER]: Becoming Leader. State: Replica: 73a1f2ca1f544de9b0e3b37015f85328, State: Running, Role: LEADER
I20240502 21:29:27.699422 24532 consensus_queue.cc:237] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:29:27.718231 24537 raft_consensus.cc:482] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.718389 24537 raft_consensus.cc:504] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:29:27.719098 24537 leader_election.cc:288] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:29:27.719712 24515 raft_consensus.cc:482] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.719892 24515 raft_consensus.cc:504] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:27.720491 24515 leader_election.cc:288] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:29:27.722249 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b589c7da65e34002b56cbf867d033c76" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:29:27.722414 7484 raft_consensus.cc:2454] T b589c7da65e34002b56cbf867d033c76 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
W20240502 21:29:27.724371 7148 leader_election.cc:334] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
W20240502 21:29:27.725206 7416 leader_election.cc:334] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
W20240502 21:29:27.725441 7420 leader_election.cc:334] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:27.725543 7420 leader_election.cc:302] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a72286397c4edc9a63eca3412be168; no voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:27.726037 7148 leader_election.cc:302] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 39981a8dfe94462b9a73ba489af4c714; no voters: 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:27.726456 24515 raft_consensus.cc:2781] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:29:27.726542 24515 raft_consensus.cc:482] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.726615 24515 raft_consensus.cc:3037] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:29:27.728821 7552 consensus_peers.cc:485] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:27.729504 24515 raft_consensus.cc:504] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:27.729888 24032 raft_consensus.cc:2726] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:27.730715 24515 leader_election.cc:288] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 2 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:29:27.733062 24527 raft_consensus.cc:482] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.733239 7484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b589c7da65e34002b56cbf867d033c76" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:29:27.733264 24527 raft_consensus.cc:504] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:29:27.733405 7484 raft_consensus.cc:3037] T b589c7da65e34002b56cbf867d033c76 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:29:27.733639 7148 leader_election.cc:334] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.733937 24527 leader_election.cc:288] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:29:27.734935 7347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bddb5e40d464de082c1d586618c86d7" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
I20240502 21:29:27.735127 7347 raft_consensus.cc:2454] T 1bddb5e40d464de082c1d586618c86d7 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:29:27.735503 7551 leader_election.cc:302] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, d0456c5e55a542128688904e003d403c; no voters:
W20240502 21:29:27.736580 7549 leader_election.cc:334] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.736905 24527 raft_consensus.cc:2781] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:29:27.736995 24527 raft_consensus.cc:482] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:27.737075 24527 raft_consensus.cc:3037] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:27.737733 7484 raft_consensus.cc:2454] T b589c7da65e34002b56cbf867d033c76 P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 2.
I20240502 21:29:27.738149 7148 leader_election.cc:302] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, 39981a8dfe94462b9a73ba489af4c714; no voters: 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:27.738554 24515 raft_consensus.cc:2781] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:27.738898 24515 raft_consensus.cc:686] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [term 2 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:29:27.739264 24515 consensus_queue.cc:237] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:27.743348 4299 catalog_manager.cc:5484] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328 reported cstate change: term changed from 1 to 2, leader changed from 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) to 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68). New cstate: current_term: 2 leader_uuid: "73a1f2ca1f544de9b0e3b37015f85328" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
I20240502 21:29:27.747606 24527 raft_consensus.cc:504] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
W20240502 21:29:27.754021 7549 leader_election.cc:334] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:27.757623 24527 leader_election.cc:288] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 election: Requested vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
W20240502 21:29:27.759382 7552 consensus_peers.cc:485] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:27.759517 7285 consensus_peers.cc:462] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:27.759783 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bddb5e40d464de082c1d586618c86d7" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c"
I20240502 21:29:27.759995 7349 raft_consensus.cc:3037] T 1bddb5e40d464de082c1d586618c86d7 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:27.762048 7349 raft_consensus.cc:2454] T 1bddb5e40d464de082c1d586618c86d7 P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 2.
W20240502 21:29:27.762061 4773 consensus_peers.cc:485] T 5f614ef09d764740926c10618370084f P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:27.762413 7551 leader_election.cc:302] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, d0456c5e55a542128688904e003d403c; no voters: 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:27.763763 24527 raft_consensus.cc:2781] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:27.763913 24527 raft_consensus.cc:686] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 [term 2 LEADER]: Becoming Leader. State: Replica: 0ddd486109f5434cace13d60d633ac04, State: Running, Role: LEADER
I20240502 21:29:27.764159 24527 consensus_queue.cc:237] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:29:27.766119 4299 catalog_manager.cc:5484] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 reported cstate change: term changed from 1 to 2, leader changed from 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) to 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69). New cstate: current_term: 2 leader_uuid: "39981a8dfe94462b9a73ba489af4c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } }
I20240502 21:29:27.770705 4292 catalog_manager.cc:5484] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 reported cstate change: term changed from 1 to 2, leader changed from 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65) to 0ddd486109f5434cace13d60d633ac04 (127.29.82.72). New cstate: current_term: 2 leader_uuid: "0ddd486109f5434cace13d60d633ac04" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } }
W20240502 21:29:27.775707 4511 consensus_peers.cc:485] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:27.777966 7213 raft_consensus.cc:1261] T 0f822150741f4ec1a3a9472572160359 P 39981a8dfe94462b9a73ba489af4c714 [term 2 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:29:27.778811 24537 consensus_queue.cc:1035] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:29:27.796125 7420 consensus_peers.cc:485] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:27.801882 4573 raft_consensus.cc:1261] T 2285212d78b0460ca329949eb3140e69 P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 2 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:29:27.806317 24527 consensus_queue.cc:1035] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:27.835259 4837 raft_consensus.cc:1261] T 0914e3ce5a2d43788ba9d369227e4f66 P 73a1f2ca1f544de9b0e3b37015f85328 [term 2 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:29:27.836885 24530 consensus_queue.cc:1035] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:27.840693 7349 raft_consensus.cc:1261] T 2ae670e3d8ae49f2b8c7df65b45b179e P d0456c5e55a542128688904e003d403c [term 3 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 21:29:27.841580 24530 consensus_queue.cc:1035] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 21:29:27.848505 7552 consensus_peers.cc:485] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:27.850451 7549 consensus_peers.cc:485] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:27.872256 7549 consensus_peers.cc:485] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:27.892515 7416 consensus_peers.cc:485] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:27.899891 7618 raft_consensus.cc:1261] T a4c36adfc86141fbb8724d5a6a798c26 P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:29:27.900803 24537 consensus_queue.cc:1035] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:29:27.930183 7420 consensus_peers.cc:485] T a4c36adfc86141fbb8724d5a6a798c26 P 31a72286397c4edc9a63eca3412be168 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:27.963838 4574 raft_consensus.cc:1261] T 38ce97f9f9ac4398a16958931ef7262f P 7a736a8ebb164ce0a051c0fccd76c3b8 [term 2 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:29:27.965847 24568 consensus_queue.cc:1035] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:27.981757 4837 raft_consensus.cc:1261] T a7a0167853fa420586455b82fb893c46 P 73a1f2ca1f544de9b0e3b37015f85328 [term 2 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:29:27.982451 24525 consensus_queue.cc:1035] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:27.985157 7213 raft_consensus.cc:1261] T 3a1fadb11cd6436a94e4a67a3aad17ba P 39981a8dfe94462b9a73ba489af4c714 [term 2 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:29:27.985709 24555 consensus_queue.cc:1035] T 3a1fadb11cd6436a94e4a67a3aad17ba P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 21:29:28.009574 7549 consensus_peers.cc:485] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:28.024507 4837 raft_consensus.cc:1261] T fb2cc722452f4d9798353d1bfae4bd4f P 73a1f2ca1f544de9b0e3b37015f85328 [term 2 FOLLOWER]: Refusing update from remote peer 31a72286397c4edc9a63eca3412be168: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:29:28.025022 24537 consensus_queue.cc:1035] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:28.082310 7349 raft_consensus.cc:1261] T ca332df7aaef4ddd9fd3266e8003b55f P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:29:28.083158 24568 consensus_queue.cc:1035] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:29:28.087433 4770 consensus_peers.cc:485] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:28.101424 7347 raft_consensus.cc:1261] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:29:28.102259 24555 consensus_queue.cc:1035] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 21:29:28.111117 4508 consensus_peers.cc:485] T 14bd28b4c6a54d98a18351b60f299a3a P 7a736a8ebb164ce0a051c0fccd76c3b8 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:28.111909 7482 raft_consensus.cc:1261] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:29:28.112480 24555 consensus_queue.cc:1035] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:29:28.120474 4508 consensus_peers.cc:485] T 2eb73e73b64b4b0f9e5116f258f2e147 P 7a736a8ebb164ce0a051c0fccd76c3b8 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:28.121861 7482 raft_consensus.cc:1261] T b589c7da65e34002b56cbf867d033c76 P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Refusing update from remote peer 39981a8dfe94462b9a73ba489af4c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:29:28.122464 24150 consensus_queue.cc:1035] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:29:28.128502 7148 consensus_peers.cc:485] T b589c7da65e34002b56cbf867d033c76 P 39981a8dfe94462b9a73ba489af4c714 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:28.139353 4508 consensus_peers.cc:485] T 5d710d1340514764b0c7d38e6bf19436 P 7a736a8ebb164ce0a051c0fccd76c3b8 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:28.198910 7148 consensus_peers.cc:462] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1203: Failure
Value of: s.ok()
Actual: false
Expected: true
Runtime error: /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 40
Maximum Replica Count | 52
Average Replica Count | 48.750000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 1
Maximum | 3
Average | 2.636364
stderr: W20240502 21:27:29.207796 9463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:29.208190 9463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:29.217242 9463 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:27:30.887902 9463 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 21:27:31.830096 9463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 569b30a95e114b50ada524f1380f198e: could not find any suitable replica to move from server 39981a8dfe94462b9a73ba489af4c714 to server 73a1f2ca1f544de9b0e3b37015f85328
W20240502 21:27:31.830406 9463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ccfdd338c3a14e4eaab745d78ee6de47: could not find any suitable replica to move from server 73a1f2ca1f544de9b0e3b37015f85328 to server 31a72286397c4edc9a63eca3412be168
W20240502 21:27:31.830644 9463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 943341d7da6a4071b6595f437f13e626: could not find any suitable replica to move from server 7a736a8ebb164ce0a051c0fccd76c3b8 to server 39981a8dfe94462b9a73ba489af4c714
W20240502 21:27:31.830889 9463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57ddb6dad3c444239d35a1ed0119bba9: could not find any suitable replica to move from server 0ddd486109f5434cace13d60d633ac04 to server 31a72286397c4edc9a63eca3412be168
W20240502 21:27:31.831269 9463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6578c35615d249838e00b267cf4291d8: could not find any suitable replica to move from server 398c3c8f3195492c900439ad5e1931bb to server 39981a8dfe94462b9a73ba489af4c714
W20240502 21:27:31.832113 9463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9606abae682a4594ada645db2e56a7d4: could not find any suitable replica to move from server d0456c5e55a542128688904e003d403c to server 7a736a8ebb164ce0a051c0fccd76c3b8
W20240502 21:27:31.832301 9463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0b924d712d1446f6b103821acdc47c7b: could not find any suitable replica to move from server 7a736a8ebb164ce0a051c0fccd76c3b8 to server 0ddd486109f5434cace13d60d633ac04
W20240502 21:27:31.832653 9463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d6c511862aad4ba5a222ddedb7e95c27: could not find any suitable replica to move from server 7a736a8ebb164ce0a051c0fccd76c3b8 to server d0456c5e55a542128688904e003d403c
W20240502 21:27:31.832969 9463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 13f15c7aba6346b19da83861cfdb5f07: could not find any suitable replica to move from server 73a1f2ca1f544de9b0e3b37015f85328 to server 0ddd486109f5434cace13d60d633ac04
W20240502 21:27:31.833257 9463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d14fe34dcad44629af56956cc9e71e5b: could not find any suitable replica to move from server 7a736a8ebb164ce0a051c0fccd76c3b8 to server 0ddd486109f5434cace13d60d633ac04
W20240502 21:27:31.833469 9463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 23c1b78fb75948c193bd4eb1c32f8bef: could not find any suitable replica to move from server d0456c5e55a542128688904e003d403c to server 71e1375b799b439fa3c83ca763c039c1
W20240502 21:27:31.833796 9463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6a89f5df3f6a4844a9c775290e276fa0: could not find any suitable replica to move from server 39981a8dfe94462b9a73ba489af4c714 to server 71e1375b799b439fa3c83ca763c039c1
W20240502 21:27:31.834520 9463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 50e84343189b44c5bd6158fbd7ba5394: could not find any suitable replica to move from server 7a736a8ebb164ce0a051c0fccd76c3b8 to server 73a1f2ca1f544de9b0e3b37015f85328
W20240502 21:27:31.834648 9463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1488a4e68a204d09a88b4a05c1c2cc24: could not find any suitable replica to move from server 0ddd486109f5434cace13d60d633ac04 to server d0456c5e55a542128688904e003d403c
W20240502 21:27:31.834731 9463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cc837d2089e24e0fa0a7c398ca3eeca8: could not find any suitable replica to move from server 73a1f2ca1f544de9b0e3b37015f85328 to server 31a72286397c4edc9a63eca3412be168
W20240502 21:27:31.834856 9463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e508630eb48f45f88993a4b8e7fd2ea6: could not find any suitable replica to move from server d0456c5e55a542128688904e003d403c to server 73a1f2ca1f544de9b0e3b37015f85328
W20240502 21:27:31.835005 9463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0b924d712d1446f6b103821acdc47c7b: could not find any suitable replica to move from server 73a1f2ca1f544de9b0e3b37015f85328 to server 39981a8dfe94462b9a73ba489af4c714
W20240502 21:27:31.835144 9463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d6c511862aad4ba5a222ddedb7e95c27: could not find any suitable replica to move from server 73a1f2ca1f544de9b0e3b37015f85328 to server 39981a8dfe94462b9a73ba489af4c714
W20240502 21:27:31.835330 9463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 80b493b0fc3f49dcbf948cee9e748e3c: could not find any suitable replica to move from server 73a1f2ca1f544de9b0e3b37015f85328 to server 31a72286397c4edc9a63eca3412be168
W20240502 21:27:31.835462 9463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 23c1b78fb75948c193bd4eb1c32f8bef: could not find any suitable replica to move from server 73a1f2ca1f544de9b0e3b37015f85328 to server 39981a8dfe94462b9a73ba489af4c714
W20240502 21:27:31.835551 9463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table add5d1abdc46419a86ab49c59629bc2d: could not find any suitable replica to move from server 0ddd486109f5434cace13d60d633ac04 to server 398c3c8f3195492c900439ad5e1931bb
W20240502 21:27:31.835708 9463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ccfdd338c3a14e4eaab745d78ee6de47: could not find any suitable replica to move from server 7a736a8ebb164ce0a051c0fccd76c3b8 to server d0456c5e55a542128688904e003d403c
W20240502 21:27:31.835793 9463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 943341d7da6a4071b6595f437f13e626: could not find any suitable replica to move from server 73a1f2ca1f544de9b0e3b37015f85328 to server d0456c5e55a542128688904e003d403c
W20240502 21:27:31.835896 9463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 879d91ac491a4b2dbed04af44dc5e8de: could not find any suitable replica to move from server 7a736a8ebb164ce0a051c0fccd76c3b8 to server 31a72286397c4edc9a63eca3412be168
W20240502 21:27:31.836023 9463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3c51dfaf2d824f93813d8d39758c9a8c: could not find any suitable replica to move from server 71e1375b799b439fa3c83ca763c039c1 to server 31a72286397c4edc9a63eca3412be168
W20240502 21:27:31.836365 9463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a99691f2b4cb42afb50b56d30958d085: could not find any suitable replica to move from server 31a72286397c4edc9a63eca3412be168 to server 71e1375b799b439fa3c83ca763c039c1
I20240502 21:27:31.921624 9463 rebalancer_tool.cc:1074] tablet 4c06f3bc97254bff8181d3f6f02470b4: '39981a8dfe94462b9a73ba489af4c714' -> '71e1375b799b439fa3c83ca763c039c1' move scheduled
I20240502 21:27:31.984417 9463 rebalancer_tool.cc:1074] tablet 86fa4255c6554ba28e141de3cb328976: '398c3c8f3195492c900439ad5e1931bb' -> '0ddd486109f5434cace13d60d633ac04' move scheduled
I20240502 21:27:32.076393 9463 rebalancer_tool.cc:1074] tablet 796c7f3cc1f9426daecc89add3c6aa57: '31a72286397c4edc9a63eca3412be168' -> '73a1f2ca1f544de9b0e3b37015f85328' move scheduled
I20240502 21:27:32.145293 9463 rebalancer_tool.cc:1074] tablet 030fa30553294e51808df57d6bb70b66: 'd0456c5e55a542128688904e003d403c' -> '7a736a8ebb164ce0a051c0fccd76c3b8' move scheduled
I20240502 21:27:32.215066 9463 rebalancer_tool.cc:1074] tablet e8f42cdee5c24db982392d2057b588a2: '39981a8dfe94462b9a73ba489af4c714' -> '71e1375b799b439fa3c83ca763c039c1' move scheduled
I20240502 21:27:32.328946 9463 rebalancer_tool.cc:1074] tablet c9fe2e18aa6c407fbbcbb74bf45bd520: '398c3c8f3195492c900439ad5e1931bb' -> '0ddd486109f5434cace13d60d633ac04' move scheduled
I20240502 21:27:32.492702 9463 rebalancer_tool.cc:1074] tablet 616eda3cb218475c8bf0108cc6b90810: '31a72286397c4edc9a63eca3412be168' -> '73a1f2ca1f544de9b0e3b37015f85328' move scheduled
I20240502 21:27:32.617688 9463 rebalancer_tool.cc:1074] tablet fed543e42f844227a14a8d812ce158ba: 'd0456c5e55a542128688904e003d403c' -> '7a736a8ebb164ce0a051c0fccd76c3b8' move scheduled
I20240502 21:27:32.764094 9463 rebalancer_tool.cc:1074] tablet 202f8b10d5674afa875da2f3929bd8da: '39981a8dfe94462b9a73ba489af4c714' -> '71e1375b799b439fa3c83ca763c039c1' move scheduled
I20240502 21:27:32.971556 9463 rebalancer_tool.cc:1074] tablet a7a0167853fa420586455b82fb893c46: '398c3c8f3195492c900439ad5e1931bb' -> '0ddd486109f5434cace13d60d633ac04' move scheduled
I20240502 21:27:33.098791 9463 rebalancer_tool.cc:1074] tablet 4cc82d5106804490993007f9f75943aa: '31a72286397c4edc9a63eca3412be168' -> '73a1f2ca1f544de9b0e3b37015f85328' move scheduled
I20240502 21:27:33.173916 9463 rebalancer_tool.cc:1074] tablet 37d864bed6ef46e4a3f57de0373bbe57: 'd0456c5e55a542128688904e003d403c' -> '7a736a8ebb164ce0a051c0fccd76c3b8' move scheduled
I20240502 21:27:33.250011 9463 rebalancer_tool.cc:1074] tablet 402e76e6f92744fbb36653d04f861e4f: '71e1375b799b439fa3c83ca763c039c1' -> '39981a8dfe94462b9a73ba489af4c714' move scheduled
I20240502 21:27:33.383921 9463 rebalancer_tool.cc:1074] tablet 80304ab48274408393c2e245477c3a78: '0ddd486109f5434cace13d60d633ac04' -> '398c3c8f3195492c900439ad5e1931bb' move scheduled
I20240502 21:27:33.462568 9463 rebalancer_tool.cc:1074] tablet d6370cfa3d804e0fa5dc2a0f71d78840: '31a72286397c4edc9a63eca3412be168' -> '73a1f2ca1f544de9b0e3b37015f85328' move scheduled
I20240502 21:27:33.543485 9463 rebalancer_tool.cc:1074] tablet 1c243b27429043aa8642d9b2ed595ab3: '7a736a8ebb164ce0a051c0fccd76c3b8' -> 'd0456c5e55a542128688904e003d403c' move scheduled
I20240502 21:27:33.646265 9463 rebalancer_tool.cc:1074] tablet 2285212d78b0460ca329949eb3140e69: '71e1375b799b439fa3c83ca763c039c1' -> '0ddd486109f5434cace13d60d633ac04' move scheduled
I20240502 21:27:33.704833 9463 rebalancer_tool.cc:1074] tablet 81db5e77d00446cba3eb8345ee3df476: '39981a8dfe94462b9a73ba489af4c714' -> '398c3c8f3195492c900439ad5e1931bb' move scheduled
I20240502 21:27:33.993170 9463 rebalancer_tool.cc:1074] tablet acdc893a70224a7b82083e998910d6ef: '31a72286397c4edc9a63eca3412be168' -> '7a736a8ebb164ce0a051c0fccd76c3b8' move scheduled
I20240502 21:27:34.151916 9463 rebalancer_tool.cc:1074] tablet adbccf5386624b28af36ff893ac9378f: '73a1f2ca1f544de9b0e3b37015f85328' -> 'd0456c5e55a542128688904e003d403c' move scheduled
I20240502 21:27:50.445225 9463 rebalancer_tool.cc:1136] tablet acdc893a70224a7b82083e998910d6ef: '31a72286397c4edc9a63eca3412be168' -> '7a736a8ebb164ce0a051c0fccd76c3b8' move completed: OK
I20240502 21:27:50.456252 9463 rebalancer_tool.cc:1136] tablet 2285212d78b0460ca329949eb3140e69: '71e1375b799b439fa3c83ca763c039c1' -> '0ddd486109f5434cace13d60d633ac04' move completed: OK
I20240502 21:27:50.469025 9463 rebalancer_tool.cc:1136] tablet e8f42cdee5c24db982392d2057b588a2: '39981a8dfe94462b9a73ba489af4c714' -> '71e1375b799b439fa3c83ca763c039c1' move completed: OK
I20240502 21:27:50.485677 9463 rebalancer_tool.cc:1136] tablet 030fa30553294e51808df57d6bb70b66: 'd0456c5e55a542128688904e003d403c' -> '7a736a8ebb164ce0a051c0fccd76c3b8' move completed: OK
I20240502 21:27:50.498566 9463 rebalancer_tool.cc:1136] tablet 86fa4255c6554ba28e141de3cb328976: '398c3c8f3195492c900439ad5e1931bb' -> '0ddd486109f5434cace13d60d633ac04' move completed: OK
I20240502 21:27:51.294200 9463 rebalancer_tool.cc:1136] tablet 202f8b10d5674afa875da2f3929bd8da: '39981a8dfe94462b9a73ba489af4c714' -> '71e1375b799b439fa3c83ca763c039c1' move completed: OK
I20240502 21:27:51.406924 9463 rebalancer_tool.cc:1136] tablet 796c7f3cc1f9426daecc89add3c6aa57: '31a72286397c4edc9a63eca3412be168' -> '73a1f2ca1f544de9b0e3b37015f85328' move completed: OK
I20240502 21:27:51.425989 9463 rebalancer_tool.cc:1136] tablet 4c06f3bc97254bff8181d3f6f02470b4: '39981a8dfe94462b9a73ba489af4c714' -> '71e1375b799b439fa3c83ca763c039c1' move completed: OK
I20240502 21:27:51.442658 9463 rebalancer_tool.cc:1136] tablet 616eda3cb218475c8bf0108cc6b90810: '31a72286397c4edc9a63eca3412be168' -> '73a1f2ca1f544de9b0e3b37015f85328' move completed: OK
I20240502 21:27:51.453085 9463 rebalancer_tool.cc:1136] tablet c9fe2e18aa6c407fbbcbb74bf45bd520: '398c3c8f3195492c900439ad5e1931bb' -> '0ddd486109f5434cace13d60d633ac04' move completed: OK
I20240502 21:27:51.462867 9463 rebalancer_tool.cc:1136] tablet a7a0167853fa420586455b82fb893c46: '398c3c8f3195492c900439ad5e1931bb' -> '0ddd486109f5434cace13d60d633ac04' move completed: OK
I20240502 21:27:51.472771 9463 rebalancer_tool.cc:1136] tablet adbccf5386624b28af36ff893ac9378f: '73a1f2ca1f544de9b0e3b37015f85328' -> 'd0456c5e55a542128688904e003d403c' move completed: OK
I20240502 21:27:51.482494 9463 rebalancer_tool.cc:1136] tablet 4cc82d5106804490993007f9f75943aa: '31a72286397c4edc9a63eca3412be168' -> '73a1f2ca1f544de9b0e3b37015f85328' move completed: OK
I20240502 21:27:51.496419 9463 rebalancer_tool.cc:1136] tablet 81db5e77d00446cba3eb8345ee3df476: '39981a8dfe94462b9a73ba489af4c714' -> '398c3c8f3195492c900439ad5e1931bb' move completed: OK
I20240502 21:27:51.505112 9463 rebalancer_tool.cc:1136] tablet 402e76e6f92744fbb36653d04f861e4f: '71e1375b799b439fa3c83ca763c039c1' -> '39981a8dfe94462b9a73ba489af4c714' move completed: OK
I20240502 21:27:51.518657 9463 rebalancer_tool.cc:1136] tablet d6370cfa3d804e0fa5dc2a0f71d78840: '31a72286397c4edc9a63eca3412be168' -> '73a1f2ca1f544de9b0e3b37015f85328' move completed: OK
I20240502 21:27:51.533329 9463 rebalancer_tool.cc:1136] tablet 37d864bed6ef46e4a3f57de0373bbe57: 'd0456c5e55a542128688904e003d403c' -> '7a736a8ebb164ce0a051c0fccd76c3b8' move completed: OK
I20240502 21:27:51.543352 9463 rebalancer_tool.cc:1136] tablet 80304ab48274408393c2e245477c3a78: '0ddd486109f5434cace13d60d633ac04' -> '398c3c8f3195492c900439ad5e1931bb' move completed: OK
I20240502 21:27:51.552598 9463 rebalancer_tool.cc:1136] tablet 1c243b27429043aa8642d9b2ed595ab3: '7a736a8ebb164ce0a051c0fccd76c3b8' -> 'd0456c5e55a542128688904e003d403c' move completed: OK
I20240502 21:27:51.601019 9463 rebalancer_tool.cc:1074] tablet d09a80f79473469e92310ded0f4edfdd: '31a72286397c4edc9a63eca3412be168' -> '71e1375b799b439fa3c83ca763c039c1' move scheduled
I20240502 21:27:51.661682 9463 rebalancer_tool.cc:1074] tablet a5d47492f64f49d584246fc5007281aa: '39981a8dfe94462b9a73ba489af4c714' -> '73a1f2ca1f544de9b0e3b37015f85328' move scheduled
I20240502 21:27:51.747494 9463 rebalancer_tool.cc:1074] tablet 180e178d0c714a35b8ca8843fad466c0: '0ddd486109f5434cace13d60d633ac04' -> '7a736a8ebb164ce0a051c0fccd76c3b8' move scheduled
I20240502 21:27:51.891661 9463 rebalancer_tool.cc:1074] tablet b7c60d990751440392ab6e5feb0d8498: '398c3c8f3195492c900439ad5e1931bb' -> 'd0456c5e55a542128688904e003d403c' move scheduled
I20240502 21:27:51.967823 9463 rebalancer_tool.cc:1074] tablet 9e5434cb329e4bea85317d18f6e6867d: '31a72286397c4edc9a63eca3412be168' -> '71e1375b799b439fa3c83ca763c039c1' move scheduled
I20240502 21:27:52.083341 9463 rebalancer_tool.cc:1074] tablet 3a1fadb11cd6436a94e4a67a3aad17ba: '73a1f2ca1f544de9b0e3b37015f85328' -> '39981a8dfe94462b9a73ba489af4c714' move scheduled
I20240502 21:27:52.161872 9463 rebalancer_tool.cc:1074] tablet a417562f482e4d468095fbe833056233: '0ddd486109f5434cace13d60d633ac04' -> '7a736a8ebb164ce0a051c0fccd76c3b8' move scheduled
I20240502 21:27:52.281150 9463 rebalancer_tool.cc:1074] tablet 292bfad92b14456da85bdc874fb7e06e: '398c3c8f3195492c900439ad5e1931bb' -> 'd0456c5e55a542128688904e003d403c' move scheduled
I20240502 21:27:52.423867 9463 rebalancer_tool.cc:1074] tablet 2eb73e73b64b4b0f9e5116f258f2e147: '39981a8dfe94462b9a73ba489af4c714' -> '31a72286397c4edc9a63eca3412be168' move scheduled
I20240502 21:27:52.480729 9463 rebalancer_tool.cc:1074] tablet 84c8bdb300064adc86a4b23e07d512d3: '73a1f2ca1f544de9b0e3b37015f85328' -> '71e1375b799b439fa3c83ca763c039c1' move scheduled
I20240502 21:27:52.576242 9463 rebalancer_tool.cc:1074] tablet 29235795acc147f68d2ccce93564ebd3: '0ddd486109f5434cace13d60d633ac04' -> '7a736a8ebb164ce0a051c0fccd76c3b8' move scheduled
I20240502 21:27:52.699973 9463 rebalancer_tool.cc:1074] tablet dd1cf8e447e54d6e907ffa0e4681515f: '398c3c8f3195492c900439ad5e1931bb' -> 'd0456c5e55a542128688904e003d403c' move scheduled
I20240502 21:27:52.946679 9463 rebalancer_tool.cc:1074] tablet ca332df7aaef4ddd9fd3266e8003b55f: '39981a8dfe94462b9a73ba489af4c714' -> '0ddd486109f5434cace13d60d633ac04' move scheduled
I20240502 21:27:53.086149 9463 rebalancer_tool.cc:1074] tablet 8991236419e642bfa438b9e45a1b8235: '31a72286397c4edc9a63eca3412be168' -> '398c3c8f3195492c900439ad5e1931bb' move scheduled
I20240502 21:27:53.314795 9463 rebalancer_tool.cc:1074] tablet b4298fc9f7694af19716ef4a908a501c: 'd0456c5e55a542128688904e003d403c' -> '73a1f2ca1f544de9b0e3b37015f85328' move scheduled
I20240502 21:27:53.449645 9463 rebalancer_tool.cc:1074] tablet c6459a8c391249529d2672ab6c62a8af: '71e1375b799b439fa3c83ca763c039c1' -> '7a736a8ebb164ce0a051c0fccd76c3b8' move scheduled
I20240502 21:27:53.519779 9463 rebalancer_tool.cc:1074] tablet 0914e3ce5a2d43788ba9d369227e4f66: '39981a8dfe94462b9a73ba489af4c714' -> '0ddd486109f5434cace13d60d633ac04' move scheduled
I20240502 21:27:53.614990 9463 rebalancer_tool.cc:1074] tablet 1d3be8c5eaac423db11b282739cd1d40: '31a72286397c4edc9a63eca3412be168' -> '398c3c8f3195492c900439ad5e1931bb' move scheduled
I20240502 21:28:09.212246 9463 rebalancer_tool.cc:1136] tablet fed543e42f844227a14a8d812ce158ba: 'd0456c5e55a542128688904e003d403c' -> '7a736a8ebb164ce0a051c0fccd76c3b8' move completed: OK
I20240502 21:28:09.263518 9463 rebalancer_tool.cc:1074] tablet fb2cc722452f4d9798353d1bfae4bd4f: 'd0456c5e55a542128688904e003d403c' -> '73a1f2ca1f544de9b0e3b37015f85328' move scheduled
I20240502 21:28:09.344444 9463 rebalancer_tool.cc:1074] tablet fd610e340cd3409397d8f23b623b4074: '7a736a8ebb164ce0a051c0fccd76c3b8' -> '71e1375b799b439fa3c83ca763c039c1' move scheduled
I20240502 21:28:11.655459 9463 rebalancer_tool.cc:1136] tablet c6459a8c391249529d2672ab6c62a8af: '71e1375b799b439fa3c83ca763c039c1' -> '7a736a8ebb164ce0a051c0fccd76c3b8' move completed: OK
I20240502 21:28:11.671391 9463 rebalancer_tool.cc:1136] tablet b4298fc9f7694af19716ef4a908a501c: 'd0456c5e55a542128688904e003d403c' -> '73a1f2ca1f544de9b0e3b37015f85328' move completed: OK
I20240502 21:28:11.680277 9463 rebalancer_tool.cc:1136] tablet 8991236419e642bfa438b9e45a1b8235: '31a72286397c4edc9a63eca3412be168' -> '398c3c8f3195492c900439ad5e1931bb' move completed: OK
I20240502 21:28:11.692437 9463 rebalancer_tool.cc:1136] tablet dd1cf8e447e54d6e907ffa0e4681515f: '398c3c8f3195492c900439ad5e1931bb' -> 'd0456c5e55a542128688904e003d403c' move completed: OK
I20240502 21:28:11.704325 9463 rebalancer_tool.cc:1136] tablet 2eb73e73b64b4b0f9e5116f258f2e147: '39981a8dfe94462b9a73ba489af4c714' -> '31a72286397c4edc9a63eca3412be168' move completed: OK
I20240502 21:28:11.714092 9463 rebalancer_tool.cc:1136] tablet 292bfad92b14456da85bdc874fb7e06e: '398c3c8f3195492c900439ad5e1931bb' -> 'd0456c5e55a542128688904e003d403c' move completed: OK
I20240502 21:28:11.723052 9463 rebalancer_tool.cc:1136] tablet a417562f482e4d468095fbe833056233: '0ddd486109f5434cace13d60d633ac04' -> '7a736a8ebb164ce0a051c0fccd76c3b8' move completed: OK
I20240502 21:28:11.732447 9463 rebalancer_tool.cc:1136] tablet 3a1fadb11cd6436a94e4a67a3aad17ba: '73a1f2ca1f544de9b0e3b37015f85328' -> '39981a8dfe94462b9a73ba489af4c714' move completed: OK
I20240502 21:28:11.741386 9463 rebalancer_tool.cc:1136] tablet 9e5434cb329e4bea85317d18f6e6867d: '31a72286397c4edc9a63eca3412be168' -> '71e1375b799b439fa3c83ca763c039c1' move completed: OK
I20240502 21:28:12.497828 9463 rebalancer_tool.cc:1136] tablet 180e178d0c714a35b8ca8843fad466c0: '0ddd486109f5434cace13d60d633ac04' -> '7a736a8ebb164ce0a051c0fccd76c3b8' move completed: OK
I20240502 21:28:13.198158 9463 rebalancer_tool.cc:1136] tablet 0914e3ce5a2d43788ba9d369227e4f66: '39981a8dfe94462b9a73ba489af4c714' -> '0ddd486109f5434cace13d60d633ac04' move completed: OK
I20240502 21:28:13.214586 9463 rebalancer_tool.cc:1136] tablet 84c8bdb300064adc86a4b23e07d512d3: '73a1f2ca1f544de9b0e3b37015f85328' -> '71e1375b799b439fa3c83ca763c039c1' move completed: OK
I20240502 21:28:13.225090 9463 rebalancer_tool.cc:1136] tablet d09a80f79473469e92310ded0f4edfdd: '31a72286397c4edc9a63eca3412be168' -> '71e1375b799b439fa3c83ca763c039c1' move completed: OK
I20240502 21:28:13.236977 9463 rebalancer_tool.cc:1136] tablet ca332df7aaef4ddd9fd3266e8003b55f: '39981a8dfe94462b9a73ba489af4c714' -> '0ddd486109f5434cace13d60d633ac04' move completed: OK
I20240502 21:28:14.179148 9463 rebalancer_tool.cc:1074] tablet be423032bc4b459b8e8e63a2379e37ac: '0ddd486109f5434cace13d60d633ac04' -> '31a72286397c4edc9a63eca3412be168' move scheduled
I20240502 21:28:14.282254 9463 rebalancer_tool.cc:1074] tablet 42edb2b3ff094c71872fde7949b61b3b: '71e1375b799b439fa3c83ca763c039c1' -> '398c3c8f3195492c900439ad5e1931bb' move scheduled
I20240502 21:28:14.352443 9463 rebalancer_tool.cc:1074] tablet 831d24800da54da9b22f9e1bca474755: '39981a8dfe94462b9a73ba489af4c714' -> 'd0456c5e55a542128688904e003d403c' move scheduled
I20240502 21:28:14.429239 9463 rebalancer_tool.cc:1074] tablet 104c676bff964b8b8566b2200ce238c7: '7a736a8ebb164ce0a051c0fccd76c3b8' -> '0ddd486109f5434cace13d60d633ac04' move scheduled
I20240502 21:28:14.496899 9463 rebalancer_tool.cc:1074] tablet cfceb76b56924833bd7e6b3901573ee7: '398c3c8f3195492c900439ad5e1931bb' -> '73a1f2ca1f544de9b0e3b37015f85328' move scheduled
I20240502 21:28:14.544750 9463 rebalancer_tool.cc:1074] tablet a507a60fb2094bd3b9dc24bec06db867: '31a72286397c4edc9a63eca3412be168' -> 'd0456c5e55a542128688904e003d403c' move scheduled
I20240502 21:28:14.608783 9463 rebalancer_tool.cc:1074] tablet 38ce97f9f9ac4398a16958931ef7262f: '71e1375b799b439fa3c83ca763c039c1' -> '0ddd486109f5434cace13d60d633ac04' move scheduled
I20240502 21:28:14.696117 9463 rebalancer_tool.cc:1074] tablet f800bfd495a94f2a952bc57a348304d0: '39981a8dfe94462b9a73ba489af4c714' -> '7a736a8ebb164ce0a051c0fccd76c3b8' move scheduled
I20240502 21:28:14.878476 9463 rebalancer_tool.cc:1074] tablet d558bcaf20bd46548936d5fb946d7710: 'd0456c5e55a542128688904e003d403c' -> '73a1f2ca1f544de9b0e3b37015f85328' move scheduled
I20240502 21:28:15.207696 9463 rebalancer_tool.cc:1074] tablet 2ae670e3d8ae49f2b8c7df65b45b179e: '31a72286397c4edc9a63eca3412be168' -> '398c3c8f3195492c900439ad5e1931bb' move scheduled
I20240502 21:28:15.288432 9463 rebalancer_tool.cc:1074] tablet 61e18d2b13644f6a99225c106c594914: '0ddd486109f5434cace13d60d633ac04' -> '71e1375b799b439fa3c83ca763c039c1' move scheduled
I20240502 21:28:15.454739 9463 rebalancer_tool.cc:1074] tablet 93e0911f163f4629a4df0249c5eb8860: '39981a8dfe94462b9a73ba489af4c714' -> '7a736a8ebb164ce0a051c0fccd76c3b8' move scheduled
I20240502 21:28:24.495931 9463 rebalancer_tool.cc:1136] tablet fb2cc722452f4d9798353d1bfae4bd4f: 'd0456c5e55a542128688904e003d403c' -> '73a1f2ca1f544de9b0e3b37015f85328' move completed: OK
I20240502 21:28:24.518488 9463 rebalancer_tool.cc:1136] tablet 1d3be8c5eaac423db11b282739cd1d40: '31a72286397c4edc9a63eca3412be168' -> '398c3c8f3195492c900439ad5e1931bb' move completed: OK
I20240502 21:28:24.528903 9463 rebalancer_tool.cc:1136] tablet b7c60d990751440392ab6e5feb0d8498: '398c3c8f3195492c900439ad5e1931bb' -> 'd0456c5e55a542128688904e003d403c' move completed: OK
I20240502 21:28:27.515388 9463 rebalancer_tool.cc:1136] tablet a5d47492f64f49d584246fc5007281aa: '39981a8dfe94462b9a73ba489af4c714' -> '73a1f2ca1f544de9b0e3b37015f85328' move completed: OK
I20240502 21:28:27.526103 9463 rebalancer_tool.cc:1136] tablet 29235795acc147f68d2ccce93564ebd3: '0ddd486109f5434cace13d60d633ac04' -> '7a736a8ebb164ce0a051c0fccd76c3b8' move completed: OK
I20240502 21:28:27.592561 9463 rebalancer_tool.cc:1074] tablet 20b714b400824ecf86890f92aeff19be: '73a1f2ca1f544de9b0e3b37015f85328' -> '0ddd486109f5434cace13d60d633ac04' move scheduled
I20240502 21:28:27.647442 9463 rebalancer_tool.cc:1074] tablet 1d0fcf18fd154a72a3bb2f0744febfc7: '398c3c8f3195492c900439ad5e1931bb' -> '31a72286397c4edc9a63eca3412be168' move scheduled
I20240502 21:28:27.704703 9463 rebalancer_tool.cc:1074] tablet 0f822150741f4ec1a3a9472572160359: 'd0456c5e55a542128688904e003d403c' -> '39981a8dfe94462b9a73ba489af4c714' move scheduled
I20240502 21:28:27.772542 9463 rebalancer_tool.cc:1074] tablet 94e66998ede944a6b00478270029ebcf: '71e1375b799b439fa3c83ca763c039c1' -> '398c3c8f3195492c900439ad5e1931bb' move scheduled
I20240502 21:28:27.889068 9463 rebalancer_tool.cc:1074] tablet e95d3785435e450688d87a0ce02d0a34: '7a736a8ebb164ce0a051c0fccd76c3b8' -> '31a72286397c4edc9a63eca3412be168' move scheduled
I20240502 21:28:32.255357 9463 rebalancer_tool.cc:1136] tablet 61e18d2b13644f6a99225c106c594914: '0ddd486109f5434cace13d60d633ac04' -> '71e1375b799b439fa3c83ca763c039c1' move completed: OK
I20240502 21:28:34.101701 9463 rebalancer_tool.cc:1136] tablet f800bfd495a94f2a952bc57a348304d0: '39981a8dfe94462b9a73ba489af4c714' -> '7a736a8ebb164ce0a051c0fccd76c3b8' move completed: OK
I20240502 21:28:34.884016 9463 rebalancer_tool.cc:1136] tablet a507a60fb2094bd3b9dc24bec06db867: '31a72286397c4edc9a63eca3412be168' -> 'd0456c5e55a542128688904e003d403c' move completed: OK
I20240502 21:28:34.921856 9463 rebalancer_tool.cc:1136] tablet cfceb76b56924833bd7e6b3901573ee7: '398c3c8f3195492c900439ad5e1931bb' -> '73a1f2ca1f544de9b0e3b37015f85328' move completed: OK
I20240502 21:28:34.932061 9463 rebalancer_tool.cc:1136] tablet d558bcaf20bd46548936d5fb946d7710: 'd0456c5e55a542128688904e003d403c' -> '73a1f2ca1f544de9b0e3b37015f85328' move completed: OK
I20240502 21:28:34.940876 9463 rebalancer_tool.cc:1136] tablet 104c676bff964b8b8566b2200ce238c7: '7a736a8ebb164ce0a051c0fccd76c3b8' -> '0ddd486109f5434cace13d60d633ac04' move completed: OK
I20240502 21:28:34.951472 9463 rebalancer_tool.cc:1136] tablet be423032bc4b459b8e8e63a2379e37ac: '0ddd486109f5434cace13d60d633ac04' -> '31a72286397c4edc9a63eca3412be168' move completed: OK
I20240502 21:28:34.959980 9463 rebalancer_tool.cc:1136] tablet fd610e340cd3409397d8f23b623b4074: '7a736a8ebb164ce0a051c0fccd76c3b8' -> '71e1375b799b439fa3c83ca763c039c1' move completed: OK
I20240502 21:28:35.941360 9463 rebalancer_tool.cc:1136] tablet 38ce97f9f9ac4398a16958931ef7262f: '71e1375b799b439fa3c83ca763c039c1' -> '0ddd486109f5434cace13d60d633ac04' move completed: OK
I20240502 21:28:36.565109 9463 rebalancer_tool.cc:1136] tablet 42edb2b3ff094c71872fde7949b61b3b: '71e1375b799b439fa3c83ca763c039c1' -> '398c3c8f3195492c900439ad5e1931bb' move completed: OK
I20240502 21:28:36.653115 9463 rebalancer_tool.cc:1074] tablet 016ad8b6874c4b629882f1243ea1fba2: '73a1f2ca1f544de9b0e3b37015f85328' -> '0ddd486109f5434cace13d60d633ac04' move scheduled
I20240502 21:28:36.725433 9463 rebalancer_tool.cc:1074] tablet 357690713cd44eb3b2a297bfd6bbc17d: 'd0456c5e55a542128688904e003d403c' -> '71e1375b799b439fa3c83ca763c039c1' move scheduled
I20240502 21:28:36.810251 9463 rebalancer_tool.cc:1074] tablet 8f579ec914e540b28a1c7c2e42e498e8: '0ddd486109f5434cace13d60d633ac04' -> '7a736a8ebb164ce0a051c0fccd76c3b8' move scheduled
I20240502 21:28:36.962883 9463 rebalancer_tool.cc:1074] tablet 546a832255ad42bebdf8eaa455ba1c3b: '73a1f2ca1f544de9b0e3b37015f85328' -> '0ddd486109f5434cace13d60d633ac04' move scheduled
I20240502 21:28:37.012897 9463 rebalancer_tool.cc:1074] tablet 845233761c2e45b0a23e25107f5461f1: 'd0456c5e55a542128688904e003d403c' -> '71e1375b799b439fa3c83ca763c039c1' move scheduled
I20240502 21:28:37.116155 9463 rebalancer_tool.cc:1074] tablet 0a5fdbfa16c946a38a90be15079bda0c: '39981a8dfe94462b9a73ba489af4c714' -> '398c3c8f3195492c900439ad5e1931bb' move scheduled
I20240502 21:28:37.172832 9463 rebalancer_tool.cc:1074] tablet 448ac663d66e4b008bfd9a6c2213fdc2: '31a72286397c4edc9a63eca3412be168' -> '7a736a8ebb164ce0a051c0fccd76c3b8' move scheduled
I20240502 21:28:37.314198 9463 rebalancer_tool.cc:1074] tablet 8ffa252784984df1bbb0ce35a26d4865: '0ddd486109f5434cace13d60d633ac04' -> '73a1f2ca1f544de9b0e3b37015f85328' move scheduled
I20240502 21:28:37.432624 9463 rebalancer_tool.cc:1074] tablet 7b5db855b5b24c8b87d65c7973f90bbb: 'd0456c5e55a542128688904e003d403c' -> '398c3c8f3195492c900439ad5e1931bb' move scheduled
I20240502 21:28:37.544824 9463 rebalancer_tool.cc:1074] tablet 5f614ef09d764740926c10618370084f: '7a736a8ebb164ce0a051c0fccd76c3b8' -> '39981a8dfe94462b9a73ba489af4c714' move scheduled
I20240502 21:28:45.818464 9463 rebalancer_tool.cc:1136] tablet 94e66998ede944a6b00478270029ebcf: '71e1375b799b439fa3c83ca763c039c1' -> '398c3c8f3195492c900439ad5e1931bb' move completed: OK
I20240502 21:28:47.547935 9463 rebalancer_tool.cc:1136] tablet 20b714b400824ecf86890f92aeff19be: '73a1f2ca1f544de9b0e3b37015f85328' -> '0ddd486109f5434cace13d60d633ac04' move completed: OK
I20240502 21:28:47.563280 9463 rebalancer_tool.cc:1136] tablet 93e0911f163f4629a4df0249c5eb8860: '39981a8dfe94462b9a73ba489af4c714' -> '7a736a8ebb164ce0a051c0fccd76c3b8' move completed: OK
I20240502 21:28:47.575177 9463 rebalancer_tool.cc:1136] tablet 2ae670e3d8ae49f2b8c7df65b45b179e: '31a72286397c4edc9a63eca3412be168' -> '398c3c8f3195492c900439ad5e1931bb' move completed: OK
I20240502 21:28:47.585678 9463 rebalancer_tool.cc:1136] tablet 1d0fcf18fd154a72a3bb2f0744febfc7: '398c3c8f3195492c900439ad5e1931bb' -> '31a72286397c4edc9a63eca3412be168' move completed: OK
I20240502 21:28:47.596621 9463 rebalancer_tool.cc:1136] tablet 0f822150741f4ec1a3a9472572160359: 'd0456c5e55a542128688904e003d403c' -> '39981a8dfe94462b9a73ba489af4c714' move completed: OK
I20240502 21:28:47.605693 9463 rebalancer_tool.cc:1136] tablet 831d24800da54da9b22f9e1bca474755: '39981a8dfe94462b9a73ba489af4c714' -> 'd0456c5e55a542128688904e003d403c' move completed: OK
I20240502 21:28:47.665130 9463 rebalancer_tool.cc:1074] tablet 269eb0e719804925961e4bcd17c18596: '0ddd486109f5434cace13d60d633ac04' -> '71e1375b799b439fa3c83ca763c039c1' move scheduled
I20240502 21:28:47.746049 9463 rebalancer_tool.cc:1074] tablet 5c5cd4328db94367b4201a49d521a145: '398c3c8f3195492c900439ad5e1931bb' -> '31a72286397c4edc9a63eca3412be168' move scheduled
I20240502 21:28:47.845598 9463 rebalancer_tool.cc:1074] tablet 26a00b5700a54672b7389e87c2a4dfbd: '39981a8dfe94462b9a73ba489af4c714' -> '398c3c8f3195492c900439ad5e1931bb' move scheduled
I20240502 21:28:47.966200 9463 rebalancer_tool.cc:1074] tablet e6f70f1bafd44613abb202a37d4c8b64: '31a72286397c4edc9a63eca3412be168' -> 'd0456c5e55a542128688904e003d403c' move scheduled
I20240502 21:28:48.009219 9463 rebalancer_tool.cc:1074] tablet a33fd7d5a8db41a39ba780e16a84f9b1: '398c3c8f3195492c900439ad5e1931bb' -> '39981a8dfe94462b9a73ba489af4c714' move scheduled
I20240502 21:28:48.144562 9463 rebalancer_tool.cc:1074] tablet eb81ce7407ab4c9e919a8a5f1b98d545: '7a736a8ebb164ce0a051c0fccd76c3b8' -> '31a72286397c4edc9a63eca3412be168' move scheduled
I20240502 21:28:51.635190 9463 rebalancer_tool.cc:1136] tablet 7b5db855b5b24c8b87d65c7973f90bbb: 'd0456c5e55a542128688904e003d403c' -> '398c3c8f3195492c900439ad5e1931bb' move completed: OK
I20240502 21:28:52.708151 9463 rebalancer_tool.cc:1136] tablet 8ffa252784984df1bbb0ce35a26d4865: '0ddd486109f5434cace13d60d633ac04' -> '73a1f2ca1f544de9b0e3b37015f85328' move completed: OK
I20240502 21:28:52.738062 9463 rebalancer_tool.cc:1136] tablet 448ac663d66e4b008bfd9a6c2213fdc2: '31a72286397c4edc9a63eca3412be168' -> '7a736a8ebb164ce0a051c0fccd76c3b8' move completed: OK
I20240502 21:28:52.750248 9463 rebalancer_tool.cc:1136] tablet 0a5fdbfa16c946a38a90be15079bda0c: '39981a8dfe94462b9a73ba489af4c714' -> '398c3c8f3195492c900439ad5e1931bb' move completed: OK
I20240502 21:28:53.601586 9463 rebalancer_tool.cc:1136] tablet 546a832255ad42bebdf8eaa455ba1c3b: '73a1f2ca1f544de9b0e3b37015f85328' -> '0ddd486109f5434cace13d60d633ac04' move completed: OK
W20240502 21:28:54.219698 20062 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.29.82.71:33913, user_credentials={real_user=slave}} blocked reactor thread for 53119.3us
I20240502 21:28:54.725975 9463 rebalancer_tool.cc:1136] tablet 8f579ec914e540b28a1c7c2e42e498e8: '0ddd486109f5434cace13d60d633ac04' -> '7a736a8ebb164ce0a051c0fccd76c3b8' move completed: OK
I20240502 21:28:54.754099 9463 rebalancer_tool.cc:1136] tablet 016ad8b6874c4b629882f1243ea1fba2: '73a1f2ca1f544de9b0e3b37015f85328' -> '0ddd486109f5434cace13d60d633ac04' move completed: OK
I20240502 21:28:54.764537 9463 rebalancer_tool.cc:1136] tablet e95d3785435e450688d87a0ce02d0a34: '7a736a8ebb164ce0a051c0fccd76c3b8' -> '31a72286397c4edc9a63eca3412be168' move completed: OK
I20240502 21:28:54.801254 9463 rebalancer_tool.cc:1136] tablet 845233761c2e45b0a23e25107f5461f1: 'd0456c5e55a542128688904e003d403c' -> '71e1375b799b439fa3c83ca763c039c1' move completed: OK
I20240502 21:28:54.813575 9463 rebalancer_tool.cc:1136] tablet 357690713cd44eb3b2a297bfd6bbc17d: 'd0456c5e55a542128688904e003d403c' -> '71e1375b799b439fa3c83ca763c039c1' move completed: OK
I20240502 21:28:54.888127 9463 rebalancer_tool.cc:1074] tablet 063bdcf9cea74d5cbe3143eb16fba6a5: '0ddd486109f5434cace13d60d633ac04' -> '73a1f2ca1f544de9b0e3b37015f85328' move scheduled
I20240502 21:28:54.956024 9463 rebalancer_tool.cc:1074] tablet 6687a533112445dc86951fc778d0725b: '7a736a8ebb164ce0a051c0fccd76c3b8' -> 'd0456c5e55a542128688904e003d403c' move scheduled
I20240502 21:28:55.018446 9463 rebalancer_tool.cc:1074] tablet d095a5e8f1f34b5ea2b6797ef06f79c6: '0ddd486109f5434cace13d60d633ac04' -> '71e1375b799b439fa3c83ca763c039c1' move scheduled
I20240502 21:28:55.103147 9463 rebalancer_tool.cc:1074] tablet ba5b4c81211647dd8383ec9484bb9c75: '0ddd486109f5434cace13d60d633ac04' -> '73a1f2ca1f544de9b0e3b37015f85328' move scheduled
I20240502 21:28:55.170058 9463 rebalancer_tool.cc:1074] tablet 686a74b8c1034112b293c4c7c6bcd99a: 'd0456c5e55a542128688904e003d403c' -> '398c3c8f3195492c900439ad5e1931bb' move scheduled
I20240502 21:28:55.233585 9463 rebalancer_tool.cc:1074] tablet 0c56940914cb4161bd258bf881f06c07: '0ddd486109f5434cace13d60d633ac04' -> '71e1375b799b439fa3c83ca763c039c1' move scheduled
I20240502 21:28:55.300477 9463 rebalancer_tool.cc:1074] tablet b5099940a48f4466aa7c713bb88d108e: '398c3c8f3195492c900439ad5e1931bb' -> '39981a8dfe94462b9a73ba489af4c714' move scheduled
I20240502 21:28:55.434031 9463 rebalancer_tool.cc:1074] tablet 6065bdba6e2840b8b368d8f92d1fa260: '31a72286397c4edc9a63eca3412be168' -> '7a736a8ebb164ce0a051c0fccd76c3b8' move scheduled
I20240502 21:28:55.560428 9463 rebalancer_tool.cc:1074] tablet a607fad427764c46b56f8963952d7613: '7a736a8ebb164ce0a051c0fccd76c3b8' -> 'd0456c5e55a542128688904e003d403c' move scheduled
I20240502 21:29:02.535027 9463 rebalancer_tool.cc:1136] tablet 5c5cd4328db94367b4201a49d521a145: '398c3c8f3195492c900439ad5e1931bb' -> '31a72286397c4edc9a63eca3412be168' move completed: OK
I20240502 21:29:03.277797 9463 rebalancer_tool.cc:1136] tablet 269eb0e719804925961e4bcd17c18596: '0ddd486109f5434cace13d60d633ac04' -> '71e1375b799b439fa3c83ca763c039c1' move completed: OK
I20240502 21:29:03.290026 9463 rebalancer_tool.cc:1136] tablet 5f614ef09d764740926c10618370084f: '7a736a8ebb164ce0a051c0fccd76c3b8' -> '39981a8dfe94462b9a73ba489af4c714' move completed: OK
I20240502 21:29:03.299507 9463 rebalancer_tool.cc:1136] tablet 26a00b5700a54672b7389e87c2a4dfbd: '39981a8dfe94462b9a73ba489af4c714' -> '398c3c8f3195492c900439ad5e1931bb' move completed: OK
I20240502 21:29:04.082350 9463 rebalancer_tool.cc:1136] tablet e6f70f1bafd44613abb202a37d4c8b64: '31a72286397c4edc9a63eca3412be168' -> 'd0456c5e55a542128688904e003d403c' move completed: OK
I20240502 21:29:04.121675 9463 rebalancer_tool.cc:1136] tablet eb81ce7407ab4c9e919a8a5f1b98d545: '7a736a8ebb164ce0a051c0fccd76c3b8' -> '31a72286397c4edc9a63eca3412be168' move completed: OK
I20240502 21:29:04.196522 9463 rebalancer_tool.cc:1074] tablet db24db8916314fdc8d9c8a3b09019604: '7a736a8ebb164ce0a051c0fccd76c3b8' -> '31a72286397c4edc9a63eca3412be168' move scheduled
I20240502 21:29:04.247537 9463 rebalancer_tool.cc:1074] tablet 9a36933eaf04449faad747e5f44652b9: '0ddd486109f5434cace13d60d633ac04' -> '73a1f2ca1f544de9b0e3b37015f85328' move scheduled
I20240502 21:29:04.319425 9463 rebalancer_tool.cc:1074] tablet f4cf59f48e874f42aed76d694246e7d3: '398c3c8f3195492c900439ad5e1931bb' -> '39981a8dfe94462b9a73ba489af4c714' move scheduled
I20240502 21:29:04.428433 9463 rebalancer_tool.cc:1074] tablet a4c36adfc86141fbb8724d5a6a798c26: '7a736a8ebb164ce0a051c0fccd76c3b8' -> '31a72286397c4edc9a63eca3412be168' move scheduled
I20240502 21:29:04.500456 9463 rebalancer_tool.cc:1074] tablet 25be332e341d4869b2165da105c17a8a: 'd0456c5e55a542128688904e003d403c' -> '398c3c8f3195492c900439ad5e1931bb' move scheduled
I20240502 21:29:07.745963 9463 rebalancer_tool.cc:1136] tablet b5099940a48f4466aa7c713bb88d108e: '398c3c8f3195492c900439ad5e1931bb' -> '39981a8dfe94462b9a73ba489af4c714' move completed: OK
I20240502 21:29:07.756935 9463 rebalancer_tool.cc:1136] tablet 686a74b8c1034112b293c4c7c6bcd99a: 'd0456c5e55a542128688904e003d403c' -> '398c3c8f3195492c900439ad5e1931bb' move completed: OK
I20240502 21:29:07.768154 9463 rebalancer_tool.cc:1136] tablet ba5b4c81211647dd8383ec9484bb9c75: '0ddd486109f5434cace13d60d633ac04' -> '73a1f2ca1f544de9b0e3b37015f85328' move completed: OK
I20240502 21:29:07.791822 9463 rebalancer_tool.cc:1136] tablet d095a5e8f1f34b5ea2b6797ef06f79c6: '0ddd486109f5434cace13d60d633ac04' -> '71e1375b799b439fa3c83ca763c039c1' move completed: OK
I20240502 21:29:07.804869 9463 rebalancer_tool.cc:1136] tablet a607fad427764c46b56f8963952d7613: '7a736a8ebb164ce0a051c0fccd76c3b8' -> 'd0456c5e55a542128688904e003d403c' move completed: OK
I20240502 21:29:07.817190 9463 rebalancer_tool.cc:1136] tablet 6687a533112445dc86951fc778d0725b: '7a736a8ebb164ce0a051c0fccd76c3b8' -> 'd0456c5e55a542128688904e003d403c' move completed: OK
I20240502 21:29:07.829587 9463 rebalancer_tool.cc:1136] tablet a33fd7d5a8db41a39ba780e16a84f9b1: '398c3c8f3195492c900439ad5e1931bb' -> '39981a8dfe94462b9a73ba489af4c714' move completed: OK
I20240502 21:29:08.563287 9463 rebalancer_tool.cc:1136] tablet 063bdcf9cea74d5cbe3143eb16fba6a5: '0ddd486109f5434cace13d60d633ac04' -> '73a1f2ca1f544de9b0e3b37015f85328' move completed: OK
I20240502 21:29:08.581995 9463 rebalancer_tool.cc:1136] tablet 0c56940914cb4161bd258bf881f06c07: '0ddd486109f5434cace13d60d633ac04' -> '71e1375b799b439fa3c83ca763c039c1' move completed: OK
I20240502 21:29:08.602032 9463 rebalancer_tool.cc:1136] tablet 6065bdba6e2840b8b368d8f92d1fa260: '31a72286397c4edc9a63eca3412be168' -> '7a736a8ebb164ce0a051c0fccd76c3b8' move completed: OK
I20240502 21:29:08.664197 9463 rebalancer_tool.cc:1074] tablet 648263320b474a32bfffa18b1d4a61e3: 'd0456c5e55a542128688904e003d403c' -> '0ddd486109f5434cace13d60d633ac04' move scheduled
I20240502 21:29:08.735055 9463 rebalancer_tool.cc:1074] tablet 4dc27c0c9cee447f84e4bc1a63f64b68: '398c3c8f3195492c900439ad5e1931bb' -> '39981a8dfe94462b9a73ba489af4c714' move scheduled
I20240502 21:29:08.824232 9463 rebalancer_tool.cc:1074] tablet 128dc7838caa403282ed2868444be271: '7a736a8ebb164ce0a051c0fccd76c3b8' -> '31a72286397c4edc9a63eca3412be168' move scheduled
I20240502 21:29:08.919189 9463 rebalancer_tool.cc:1074] tablet ed822833ab3c451f96d722980b24c2b0: 'd0456c5e55a542128688904e003d403c' -> '398c3c8f3195492c900439ad5e1931bb' move scheduled
I20240502 21:29:08.976878 9463 rebalancer_tool.cc:1074] tablet 1d3528b68bc740ce9b452356fd26bfb6: '0ddd486109f5434cace13d60d633ac04' -> '39981a8dfe94462b9a73ba489af4c714' move scheduled
I20240502 21:29:09.079422 9463 rebalancer_tool.cc:1074] tablet 3394f5f087304842ada4212f2e524356: '0ddd486109f5434cace13d60d633ac04' -> '7a736a8ebb164ce0a051c0fccd76c3b8' move scheduled
I20240502 21:29:09.158596 9463 rebalancer_tool.cc:1074] tablet 6d0334be884c45c3bd9e49095454dfb1: 'd0456c5e55a542128688904e003d403c' -> '398c3c8f3195492c900439ad5e1931bb' move scheduled
I20240502 21:29:09.238188 9463 rebalancer_tool.cc:1074] tablet 2fd2328258cf4e2db0808731a86cfcf9: '39981a8dfe94462b9a73ba489af4c714' -> '0ddd486109f5434cace13d60d633ac04' move scheduled
I20240502 21:29:17.454635 9463 rebalancer_tool.cc:1136] tablet a4c36adfc86141fbb8724d5a6a798c26: '7a736a8ebb164ce0a051c0fccd76c3b8' -> '31a72286397c4edc9a63eca3412be168' move completed: OK
I20240502 21:29:17.467254 9463 rebalancer_tool.cc:1136] tablet 9a36933eaf04449faad747e5f44652b9: '0ddd486109f5434cace13d60d633ac04' -> '73a1f2ca1f544de9b0e3b37015f85328' move completed: OK
I20240502 21:29:17.480731 9463 rebalancer_tool.cc:1136] tablet db24db8916314fdc8d9c8a3b09019604: '7a736a8ebb164ce0a051c0fccd76c3b8' -> '31a72286397c4edc9a63eca3412be168' move completed: OK
I20240502 21:29:17.505887 9463 rebalancer_tool.cc:1136] tablet f4cf59f48e874f42aed76d694246e7d3: '398c3c8f3195492c900439ad5e1931bb' -> '39981a8dfe94462b9a73ba489af4c714' move completed: OK
I20240502 21:29:17.603600 9463 rebalancer_tool.cc:1074] tablet a85e7872c28a410496f31ecff9c71edb: '0ddd486109f5434cace13d60d633ac04' -> '39981a8dfe94462b9a73ba489af4c714' move scheduled
I20240502 21:29:17.653229 9463 rebalancer_tool.cc:1074] tablet 14bd28b4c6a54d98a18351b60f299a3a: '398c3c8f3195492c900439ad5e1931bb' -> 'd0456c5e55a542128688904e003d403c' move scheduled
I20240502 21:29:19.317018 9463 rebalancer_tool.cc:1136] tablet 3394f5f087304842ada4212f2e524356: '0ddd486109f5434cace13d60d633ac04' -> '7a736a8ebb164ce0a051c0fccd76c3b8' move completed: OK
I20240502 21:29:19.329157 9463 rebalancer_tool.cc:1136] tablet 1d3528b68bc740ce9b452356fd26bfb6: '0ddd486109f5434cace13d60d633ac04' -> '39981a8dfe94462b9a73ba489af4c714' move completed: OK
I20240502 21:29:19.347644 9463 rebalancer_tool.cc:1136] tablet 128dc7838caa403282ed2868444be271: '7a736a8ebb164ce0a051c0fccd76c3b8' -> '31a72286397c4edc9a63eca3412be168' move completed: OK
I20240502 21:29:19.409238 9463 rebalancer_tool.cc:1136] tablet ed822833ab3c451f96d722980b24c2b0: 'd0456c5e55a542128688904e003d403c' -> '398c3c8f3195492c900439ad5e1931bb' move completed: OK
I20240502 21:29:19.430037 9463 rebalancer_tool.cc:1136] tablet 648263320b474a32bfffa18b1d4a61e3: 'd0456c5e55a542128688904e003d403c' -> '0ddd486109f5434cace13d60d633ac04' move completed: OK
I20240502 21:29:20.133224 9463 rebalancer_tool.cc:1136] tablet 25be332e341d4869b2165da105c17a8a: 'd0456c5e55a542128688904e003d403c' -> '398c3c8f3195492c900439ad5e1931bb' move completed: OK
I20240502 21:29:20.920231 9463 rebalancer_tool.cc:1136] tablet 4dc27c0c9cee447f84e4bc1a63f64b68: '398c3c8f3195492c900439ad5e1931bb' -> '39981a8dfe94462b9a73ba489af4c714' move completed: OK
I20240502 21:29:20.980571 9463 rebalancer_tool.cc:1074] tablet 1bddb5e40d464de082c1d586618c86d7: '398c3c8f3195492c900439ad5e1931bb' -> 'd0456c5e55a542128688904e003d403c' move scheduled
I20240502 21:29:21.038064 9463 rebalancer_tool.cc:1074] tablet 81c2664e2d45448b9e08a6fac1c90a0d: '398c3c8f3195492c900439ad5e1931bb' -> '39981a8dfe94462b9a73ba489af4c714' move scheduled
I20240502 21:29:22.576122 9463 rebalancer_tool.cc:1136] tablet 6d0334be884c45c3bd9e49095454dfb1: 'd0456c5e55a542128688904e003d403c' -> '398c3c8f3195492c900439ad5e1931bb' move completed: OK
I20240502 21:29:22.586349 9463 rebalancer_tool.cc:1136] tablet 2fd2328258cf4e2db0808731a86cfcf9: '39981a8dfe94462b9a73ba489af4c714' -> '0ddd486109f5434cace13d60d633ac04' move completed: OK
I20240502 21:29:23.307814 9463 rebalancer_tool.cc:1136] tablet 14bd28b4c6a54d98a18351b60f299a3a: '398c3c8f3195492c900439ad5e1931bb' -> 'd0456c5e55a542128688904e003d403c' move completed: OK
I20240502 21:29:24.181340 9463 rebalancer_tool.cc:1136] tablet a85e7872c28a410496f31ecff9c71edb: '0ddd486109f5434cace13d60d633ac04' -> '39981a8dfe94462b9a73ba489af4c714' move completed: OK
I20240502 21:29:24.210783 9463 rebalancer_tool.cc:1136] tablet 81c2664e2d45448b9e08a6fac1c90a0d: '398c3c8f3195492c900439ad5e1931bb' -> '39981a8dfe94462b9a73ba489af4c714' move completed: OK
I20240502 21:29:24.223423 9463 rebalancer_tool.cc:1136] tablet 1bddb5e40d464de082c1d586618c86d7: '398c3c8f3195492c900439ad5e1931bb' -> 'd0456c5e55a542128688904e003d403c' move completed: OK
I20240502 21:29:25.007189 9463 rebalancer_tool.cc:1074] tablet a410b7da18004b23bdad650f59a30e8c: 'd0456c5e55a542128688904e003d403c' -> '73a1f2ca1f544de9b0e3b37015f85328' move scheduled
I20240502 21:29:25.107522 9463 rebalancer_tool.cc:1074] tablet 2285212d78b0460ca329949eb3140e69: '7a736a8ebb164ce0a051c0fccd76c3b8' -> '31a72286397c4edc9a63eca3412be168' move scheduled
I20240502 21:29:25.215240 9463 rebalancer_tool.cc:1074] tablet 686a74b8c1034112b293c4c7c6bcd99a: '73a1f2ca1f544de9b0e3b37015f85328' -> '71e1375b799b439fa3c83ca763c039c1' move scheduled
I20240502 21:29:25.260995 9463 rebalancer_tool.cc:1074] tablet c6459a8c391249529d2672ab6c62a8af: '0ddd486109f5434cace13d60d633ac04' -> '398c3c8f3195492c900439ad5e1931bb' move scheduled
I20240502 21:29:25.316617 9463 rebalancer_tool.cc:1074] tablet dd1cf8e447e54d6e907ffa0e4681515f: '7a736a8ebb164ce0a051c0fccd76c3b8' -> '39981a8dfe94462b9a73ba489af4c714' move scheduled
I20240502 21:29:25.409869 9463 rebalancer_tool.cc:1074] tablet 85fac458a4204f40a85d4ecba4718f58: 'd0456c5e55a542128688904e003d403c' -> '0ddd486109f5434cace13d60d633ac04' move scheduled
I20240502 21:29:25.496285 9463 rebalancer_tool.cc:1074] tablet 1c243b27429043aa8642d9b2ed595ab3: '398c3c8f3195492c900439ad5e1931bb' -> '31a72286397c4edc9a63eca3412be168' move scheduled
I20240502 21:29:25.595531 9463 rebalancer_tool.cc:1074] tablet 5bd324f8962647ef9e8fc01a64efe69a: '73a1f2ca1f544de9b0e3b37015f85328' -> '71e1375b799b439fa3c83ca763c039c1' move scheduled
I20240502 21:29:25.691112 9463 rebalancer_tool.cc:1074] tablet 86fa4255c6554ba28e141de3cb328976: '7a736a8ebb164ce0a051c0fccd76c3b8' -> '39981a8dfe94462b9a73ba489af4c714' move scheduled
I20240502 21:29:25.812885 9463 rebalancer_tool.cc:1074] tablet 6065bdba6e2840b8b368d8f92d1fa260: '0ddd486109f5434cace13d60d633ac04' -> 'd0456c5e55a542128688904e003d403c' move scheduled
W20240502 21:29:26.385839 24395 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 21:29:25.993801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.82.67:34537 (local address 127.0.0.1:48442)
0502 21:29:26.033225 (+ 39424us) negotiation.cc:107] Waiting for socket to connect
0502 21:29:26.033245 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0502 21:29:26.033379 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 21:29:26.384298 (+350919us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":39247,"thread_start_us":37932,"threads_started":1}
I20240502 21:29:26.390347 9463 rebalancer_tool.cc:1084] tablet 546a832255ad42bebdf8eaa455ba1c3b: '398c3c8f3195492c900439ad5e1931bb' -> '39981a8dfe94462b9a73ba489af4c714' move ignored: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:29:26.390520 9463 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): unacceptable health status UNAVAILABLE
I20240502 21:29:28.231686 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 4493
I20240502 21:29:28.238260 7350 raft_consensus.cc:1261] T 5d710d1340514764b0c7d38e6bf19436 P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Refusing update from remote peer 7a736a8ebb164ce0a051c0fccd76c3b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:29:28.238932 24278 consensus_queue.cc:1035] T 5d710d1340514764b0c7d38e6bf19436 P 7a736a8ebb164ce0a051c0fccd76c3b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:29:28.241823 7347 raft_consensus.cc:1261] T 1bddb5e40d464de082c1d586618c86d7 P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Refusing update from remote peer 0ddd486109f5434cace13d60d633ac04: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:29:28.242151 4615 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:29:28.242801 24530 consensus_queue.cc:1035] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:29:28.260092 7549 consensus_peers.cc:485] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:28.292706 7347 raft_consensus.cc:1261] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Refusing update from remote peer 73a1f2ca1f544de9b0e3b37015f85328: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:29:28.293298 23735 consensus_queue.cc:1035] T 292bfad92b14456da85bdc874fb7e06e P 73a1f2ca1f544de9b0e3b37015f85328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 21:29:28.352315 7285 consensus_peers.cc:462] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
W20240502 21:29:28.412955 4773 consensus_peers.cc:485] T eb81ce7407ab4c9e919a8a5f1b98d545 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.497090 7552 consensus_peers.cc:485] T 2707427ddc3d49b89c000e92e3b663d9 P 0ddd486109f5434cace13d60d633ac04 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.499359 7285 consensus_peers.cc:485] T 0a5fdbfa16c946a38a90be15079bda0c P d0456c5e55a542128688904e003d403c -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.519210 4773 consensus_peers.cc:485] T 7b5db855b5b24c8b87d65c7973f90bbb P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.527707 4773 consensus_peers.cc:485] T 686a74b8c1034112b293c4c7c6bcd99a P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.542971 7552 consensus_peers.cc:485] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:29:28.552944 24530 raft_consensus.cc:482] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:28.553107 24530 raft_consensus.cc:504] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:28.553781 24530 leader_election.cc:288] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
W20240502 21:29:28.555114 7552 leader_election.cc:334] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
W20240502 21:29:28.555727 7549 leader_election.cc:334] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:28.555891 7549 leader_election.cc:302] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04; no voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:28.556146 24530 raft_consensus.cc:2726] T 2fd2328258cf4e2db0808731a86cfcf9 P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:29:28.569085 7552 consensus_peers.cc:485] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.582068 7549 consensus_peers.cc:485] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.595400 4773 consensus_peers.cc:485] T 831d24800da54da9b22f9e1bca474755 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.604061 4773 consensus_peers.cc:485] T ed822833ab3c451f96d722980b24c2b0 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.613181 7285 consensus_peers.cc:485] T d714416dd87a49c6bd2d2b8de348c9df P d0456c5e55a542128688904e003d403c -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.621416 4773 consensus_peers.cc:485] T e95d3785435e450688d87a0ce02d0a34 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.624568 7151 consensus_peers.cc:485] T 94e66998ede944a6b00478270029ebcf P 39981a8dfe94462b9a73ba489af4c714 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.662704 7281 consensus_peers.cc:485] T d095a5e8f1f34b5ea2b6797ef06f79c6 P d0456c5e55a542128688904e003d403c -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.669624 7552 consensus_peers.cc:485] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.675077 7416 consensus_peers.cc:485] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.685298 7420 consensus_peers.cc:485] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:29:28.722564 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 4493
W20240502 21:29:28.726874 4773 consensus_peers.cc:485] T 104c676bff964b8b8566b2200ce238c7 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.730618 7420 consensus_peers.cc:485] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.731050 7416 consensus_peers.cc:485] T 39260f3652b9473687ce1abb9c39223e P 31a72286397c4edc9a63eca3412be168 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.755353 4773 consensus_peers.cc:485] T 9a3c115189e8442ba7c3fa7f65089d11 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.774968 7151 consensus_peers.cc:485] T 1d3be8c5eaac423db11b282739cd1d40 P 39981a8dfe94462b9a73ba489af4c714 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.781314 4770 consensus_peers.cc:485] T 202f8b10d5674afa875da2f3929bd8da P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.788964 7151 consensus_peers.cc:485] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.789009 7549 consensus_peers.cc:485] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.793392 7552 connection.cc:535] client connection to 127.29.82.66:35781 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:29:28.793473 7551 connection.cc:535] server connection from 127.29.82.66:56819 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:29:28.793592 7420 connection.cc:535] client connection to 127.29.82.66:35781 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:29:28.794931 4773 connection.cc:535] client connection to 127.29.82.66:35781 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:29:28.795010 7552 consensus_peers.cc:485] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:28.795135 7552 consensus_peers.cc:485] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 0ddd486109f5434cace13d60d633ac04 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:28.795253 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 4755
W20240502 21:29:28.795266 7552 consensus_peers.cc:485] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:28.795354 7552 consensus_peers.cc:485] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:28.795451 7552 consensus_peers.cc:485] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:28.795543 7552 consensus_peers.cc:485] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:28.796286 7285 connection.cc:535] client connection to 127.29.82.66:35781 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:29:28.796638 4773 consensus_peers.cc:485] T 1d0fcf18fd154a72a3bb2f0744febfc7 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:28.796800 4773 consensus_peers.cc:485] T 93e0911f163f4629a4df0249c5eb8860 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:28.796924 4773 consensus_peers.cc:485] T acdc893a70224a7b82083e998910d6ef P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:28.797062 4773 consensus_peers.cc:485] T a417562f482e4d468095fbe833056233 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:28.797204 4773 consensus_peers.cc:485] T dd1cf8e447e54d6e907ffa0e4681515f P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:28.797300 4773 consensus_peers.cc:485] T 86fa4255c6554ba28e141de3cb328976 P 73a1f2ca1f544de9b0e3b37015f85328 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:28.797935 7151 connection.cc:535] client connection to 127.29.82.66:35781 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:29:28.798775 7285 consensus_peers.cc:485] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:28.798928 7285 consensus_peers.cc:485] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:28.799036 7285 consensus_peers.cc:485] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:28.799149 7285 consensus_peers.cc:485] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:28.799258 7285 consensus_peers.cc:485] T 180e178d0c714a35b8ca8843fad466c0 P d0456c5e55a542128688904e003d403c -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:28.799369 7285 consensus_peers.cc:485] T 1d3528b68bc740ce9b452356fd26bfb6 P d0456c5e55a542128688904e003d403c -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:28.799538 7151 consensus_peers.cc:485] T 42edb2b3ff094c71872fde7949b61b3b P 39981a8dfe94462b9a73ba489af4c714 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:28.799676 7151 consensus_peers.cc:485] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:28.799762 7151 consensus_peers.cc:485] T 8f579ec914e540b28a1c7c2e42e498e8 P 39981a8dfe94462b9a73ba489af4c714 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:28.800045 7420 consensus_peers.cc:485] T d81b71825ba54eacb6c106bce3a838e8 P 31a72286397c4edc9a63eca3412be168 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:28.800191 7420 consensus_peers.cc:485] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:28.800266 7420 consensus_peers.cc:485] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:28.800334 7420 consensus_peers.cc:485] T f800bfd495a94f2a952bc57a348304d0 P 31a72286397c4edc9a63eca3412be168 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:28.800403 7420 consensus_peers.cc:485] T 29235795acc147f68d2ccce93564ebd3 P 31a72286397c4edc9a63eca3412be168 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:28.804535 4878 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:29:28.809455 7148 consensus_peers.cc:485] T 9e6feb60ae03404e85ef76499de190c3 P 39981a8dfe94462b9a73ba489af4c714 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.811148 7148 consensus_peers.cc:462] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
W20240502 21:29:28.813891 7420 consensus_peers.cc:485] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.829910 7285 consensus_peers.cc:462] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
W20240502 21:29:28.830094 7549 consensus_peers.cc:485] T 37d864bed6ef46e4a3f57de0373bbe57 P 0ddd486109f5434cace13d60d633ac04 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.839236 7549 consensus_peers.cc:485] T beaa535c1dbe48b48613582d65e751ae P 0ddd486109f5434cace13d60d633ac04 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.846083 7151 consensus_peers.cc:485] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.852649 7148 consensus_peers.cc:485] T 61e18d2b13644f6a99225c106c594914 P 39981a8dfe94462b9a73ba489af4c714 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:29:28.861503 24578 raft_consensus.cc:482] T 37c2c46368144212a0041e7ff4434236 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:28.861689 24578 raft_consensus.cc:504] T 37c2c46368144212a0041e7ff4434236 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:29:28.862385 24578 leader_election.cc:288] T 37c2c46368144212a0041e7ff4434236 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
W20240502 21:29:28.863893 7281 leader_election.cc:334] T 37c2c46368144212a0041e7ff4434236 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
W20240502 21:29:28.865653 7549 consensus_peers.cc:485] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.868765 7285 leader_election.cc:334] T 37c2c46368144212a0041e7ff4434236 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:28.868906 7285 leader_election.cc:302] T 37c2c46368144212a0041e7ff4434236 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0456c5e55a542128688904e003d403c; no voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:28.869186 24578 raft_consensus.cc:2726] T 37c2c46368144212a0041e7ff4434236 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:29:28.890592 7151 consensus_peers.cc:485] T 56517ab7e5e447fea86868dd0a5b3c25 P 39981a8dfe94462b9a73ba489af4c714 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.890798 7552 consensus_peers.cc:485] T d09a80f79473469e92310ded0f4edfdd P 0ddd486109f5434cace13d60d633ac04 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.911670 7148 consensus_peers.cc:485] T 612a0397b1754fda81c4a0019067a0d3 P 39981a8dfe94462b9a73ba489af4c714 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:29:28.923280 24538 raft_consensus.cc:482] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:28.923440 24538 raft_consensus.cc:504] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:29:28.923997 24538 leader_election.cc:288] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
W20240502 21:29:28.925416 7420 leader_election.cc:334] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
W20240502 21:29:28.925534 7416 leader_election.cc:334] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:28.925654 7416 leader_election.cc:302] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a72286397c4edc9a63eca3412be168; no voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:28.925868 24538 raft_consensus.cc:2726] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:29:28.944522 7552 consensus_peers.cc:485] T a8a3d57ff2184ab3bc09140afbffe0b8 P 0ddd486109f5434cace13d60d633ac04 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.975200 7281 consensus_peers.cc:485] T 84c8bdb300064adc86a4b23e07d512d3 P d0456c5e55a542128688904e003d403c -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:28.996840 7285 consensus_peers.cc:485] T 26a00b5700a54672b7389e87c2a4dfbd P d0456c5e55a542128688904e003d403c -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:29:29.005041 24578 raft_consensus.cc:482] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:29.005239 24578 raft_consensus.cc:504] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:29:29.006177 24578 leader_election.cc:288] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
W20240502 21:29:29.007616 7285 leader_election.cc:334] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
W20240502 21:29:29.008412 7281 leader_election.cc:334] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:29.008543 7281 leader_election.cc:302] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0456c5e55a542128688904e003d403c; no voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:29.008796 24578 raft_consensus.cc:2726] T a607fad427764c46b56f8963952d7613 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:29.028540 24578 raft_consensus.cc:482] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:29.028774 24578 raft_consensus.cc:504] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
W20240502 21:29:29.030825 7416 consensus_peers.cc:485] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:29:29.031277 24578 leader_election.cc:288] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
W20240502 21:29:29.032312 7285 leader_election.cc:334] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:29.032516 7482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c243b27429043aa8642d9b2ed595ab3" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:29:29.032704 7482 raft_consensus.cc:2454] T 1c243b27429043aa8642d9b2ed595ab3 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
W20240502 21:29:29.032994 7281 leader_election.cc:334] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:29.033128 7281 leader_election.cc:302] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d0456c5e55a542128688904e003d403c; no voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:29.033562 24578 raft_consensus.cc:2726] T 1c243b27429043aa8642d9b2ed595ab3 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:29:29.072629 7416 consensus_peers.cc:485] T 20b714b400824ecf86890f92aeff19be P 31a72286397c4edc9a63eca3412be168 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:29.168790 7549 consensus_peers.cc:485] T 25be332e341d4869b2165da105c17a8a P 0ddd486109f5434cace13d60d633ac04 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:29.186319 7148 consensus_peers.cc:485] T 030fa30553294e51808df57d6bb70b66 P 39981a8dfe94462b9a73ba489af4c714 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:29.186530 7281 consensus_peers.cc:485] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:29.235781 7148 consensus_peers.cc:485] T 357690713cd44eb3b2a297bfd6bbc17d P 39981a8dfe94462b9a73ba489af4c714 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:29:29.246241 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 4755
I20240502 21:29:29.248999 24608 raft_consensus.cc:482] T fed543e42f844227a14a8d812ce158ba P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a736a8ebb164ce0a051c0fccd76c3b8)
I20240502 21:29:29.249186 24608 raft_consensus.cc:504] T fed543e42f844227a14a8d812ce158ba P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:29:29.249935 24608 leader_election.cc:288] T fed543e42f844227a14a8d812ce158ba P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
W20240502 21:29:29.251442 7420 leader_election.cc:334] T fed543e42f844227a14a8d812ce158ba P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
W20240502 21:29:29.251710 7420 leader_election.cc:334] T fed543e42f844227a14a8d812ce158ba P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111)
I20240502 21:29:29.251811 7420 leader_election.cc:302] T fed543e42f844227a14a8d812ce158ba P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a72286397c4edc9a63eca3412be168; no voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:29:29.252398 24538 raft_consensus.cc:2726] T fed543e42f844227a14a8d812ce158ba P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:29:29.266888 7285 consensus_peers.cc:462] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:29.283218 24603 raft_consensus.cc:482] T 81c2664e2d45448b9e08a6fac1c90a0d P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a736a8ebb164ce0a051c0fccd76c3b8)
I20240502 21:29:29.283366 24603 raft_consensus.cc:504] T 81c2664e2d45448b9e08a6fac1c90a0d P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:29.283936 24603 leader_election.cc:288] T 81c2664e2d45448b9e08a6fac1c90a0d P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
W20240502 21:29:29.285223 7151 leader_election.cc:334] T 81c2664e2d45448b9e08a6fac1c90a0d P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111)
I20240502 21:29:29.306270 24603 raft_consensus.cc:482] T f4cf59f48e874f42aed76d694246e7d3 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a736a8ebb164ce0a051c0fccd76c3b8)
I20240502 21:29:29.306440 24603 raft_consensus.cc:504] T f4cf59f48e874f42aed76d694246e7d3 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:29.307035 24603 leader_election.cc:288] T f4cf59f48e874f42aed76d694246e7d3 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
W20240502 21:29:29.308347 7151 leader_election.cc:334] T f4cf59f48e874f42aed76d694246e7d3 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111)
W20240502 21:29:29.313297 7148 connection.cc:535] client connection to 127.29.82.68:42449 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:29:29.313287 7549 connection.cc:535] client connection to 127.29.82.68:42449 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:29:29.313427 7416 connection.cc:535] client connection to 127.29.82.68:42449 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:29:29.313778 7549 connection.cc:535] server connection from 127.29.82.68:47485 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:29:29.314119 7281 connection.cc:535] client connection to 127.29.82.68:42449 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:29:29.314419 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 7129
W20240502 21:29:29.315412 7281 consensus_peers.cc:485] T 448ac663d66e4b008bfd9a6c2213fdc2 P d0456c5e55a542128688904e003d403c -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:29.315543 7281 consensus_peers.cc:485] T b7c60d990751440392ab6e5feb0d8498 P d0456c5e55a542128688904e003d403c -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:29.315541 7549 consensus_peers.cc:485] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:29.315687 7281 consensus_peers.cc:485] T 9a36933eaf04449faad747e5f44652b9 P d0456c5e55a542128688904e003d403c -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:29.315712 7549 consensus_peers.cc:485] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04 -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:29.315810 7549 consensus_peers.cc:485] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:29.315899 7549 consensus_peers.cc:485] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:29.315984 7549 consensus_peers.cc:485] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04 -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:29.316030 7416 consensus_peers.cc:485] T b4298fc9f7694af19716ef4a908a501c P 31a72286397c4edc9a63eca3412be168 -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:29.316154 7416 consensus_peers.cc:485] T ba5b4c81211647dd8383ec9484bb9c75 P 31a72286397c4edc9a63eca3412be168 -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:29.316217 7416 consensus_peers.cc:485] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:29.316282 7416 consensus_peers.cc:485] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:29.316947 7418 connection.cc:535] server connection from 127.29.82.68:47963 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:29:29.317767 7148 leader_election.cc:334] T 81c2664e2d45448b9e08a6fac1c90a0d P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111)
I20240502 21:29:29.317883 7148 leader_election.cc:302] T 81c2664e2d45448b9e08a6fac1c90a0d P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714; no voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8
W20240502 21:29:29.318207 7148 consensus_peers.cc:485] T a507a60fb2094bd3b9dc24bec06db867 P 39981a8dfe94462b9a73ba489af4c714 -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:29.318310 7148 leader_election.cc:334] T f4cf59f48e874f42aed76d694246e7d3 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111)
I20240502 21:29:29.318403 7148 leader_election.cc:302] T f4cf59f48e874f42aed76d694246e7d3 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39981a8dfe94462b9a73ba489af4c714; no voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:29:29.318445 24603 raft_consensus.cc:2726] T 81c2664e2d45448b9e08a6fac1c90a0d P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:29.318660 24150 raft_consensus.cc:2726] T f4cf59f48e874f42aed76d694246e7d3 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:29:29.318940 7148 consensus_peers.cc:485] T 4cc82d5106804490993007f9f75943aa P 39981a8dfe94462b9a73ba489af4c714 -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:29.319061 7148 consensus_peers.cc:485] T a410b7da18004b23bdad650f59a30e8c P 39981a8dfe94462b9a73ba489af4c714 -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:29.319186 7148 consensus_peers.cc:485] T 5bd324f8962647ef9e8fc01a64efe69a P 39981a8dfe94462b9a73ba489af4c714 -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:29.319293 7148 consensus_peers.cc:485] T e6f70f1bafd44613abb202a37d4c8b64 P 39981a8dfe94462b9a73ba489af4c714 -> Peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Couldn't send request to peer 73a1f2ca1f544de9b0e3b37015f85328. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:29.328362 7256 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:29:29.334911 24530 raft_consensus.cc:482] T 773593a488a74091ac74b33af07227a6 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a736a8ebb164ce0a051c0fccd76c3b8)
I20240502 21:29:29.335093 24530 raft_consensus.cc:504] T 773593a488a74091ac74b33af07227a6 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:29:29.335769 24530 leader_election.cc:288] T 773593a488a74091ac74b33af07227a6 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
W20240502 21:29:29.338847 7549 leader_election.cc:334] T 773593a488a74091ac74b33af07227a6 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111)
W20240502 21:29:29.338994 7552 leader_election.cc:334] T 773593a488a74091ac74b33af07227a6 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111)
I20240502 21:29:29.339138 7552 leader_election.cc:302] T 773593a488a74091ac74b33af07227a6 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04; no voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:29:29.339385 24530 raft_consensus.cc:2726] T 773593a488a74091ac74b33af07227a6 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:29.342010 24608 raft_consensus.cc:482] T 9d279c5db74f46519c35fe75a35589c7 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a736a8ebb164ce0a051c0fccd76c3b8)
I20240502 21:29:29.342170 24608 raft_consensus.cc:504] T 9d279c5db74f46519c35fe75a35589c7 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:29.342820 24608 leader_election.cc:288] T 9d279c5db74f46519c35fe75a35589c7 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
I20240502 21:29:29.343307 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d279c5db74f46519c35fe75a35589c7" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0456c5e55a542128688904e003d403c" is_pre_election: true
W20240502 21:29:29.344053 7420 leader_election.cc:334] T 9d279c5db74f46519c35fe75a35589c7 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111)
I20240502 21:29:29.344156 7420 leader_election.cc:302] T 9d279c5db74f46519c35fe75a35589c7 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a72286397c4edc9a63eca3412be168; no voters: 7a736a8ebb164ce0a051c0fccd76c3b8, d0456c5e55a542128688904e003d403c
I20240502 21:29:29.344374 24608 raft_consensus.cc:2726] T 9d279c5db74f46519c35fe75a35589c7 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:29.350257 24608 raft_consensus.cc:482] T ebcf1894471f4fb7b951a730c2a781dd P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a736a8ebb164ce0a051c0fccd76c3b8)
I20240502 21:29:29.350399 24608 raft_consensus.cc:504] T ebcf1894471f4fb7b951a730c2a781dd P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:29:29.350972 24608 leader_election.cc:288] T ebcf1894471f4fb7b951a730c2a781dd P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
W20240502 21:29:29.352216 7420 leader_election.cc:334] T ebcf1894471f4fb7b951a730c2a781dd P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111)
W20240502 21:29:29.352416 7416 leader_election.cc:334] T ebcf1894471f4fb7b951a730c2a781dd P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:29.352572 7416 leader_election.cc:302] T ebcf1894471f4fb7b951a730c2a781dd P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a72286397c4edc9a63eca3412be168; no voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:29:29.352818 24608 raft_consensus.cc:2726] T ebcf1894471f4fb7b951a730c2a781dd P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:29.361812 24608 raft_consensus.cc:482] T 128dc7838caa403282ed2868444be271 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:29.361968 24608 raft_consensus.cc:504] T 128dc7838caa403282ed2868444be271 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:29.362514 24608 leader_election.cc:288] T 128dc7838caa403282ed2868444be271 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
W20240502 21:29:29.363672 7416 leader_election.cc:334] T 128dc7838caa403282ed2868444be271 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
W20240502 21:29:29.363919 7420 leader_election.cc:334] T 128dc7838caa403282ed2868444be271 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:29.364046 7420 leader_election.cc:302] T 128dc7838caa403282ed2868444be271 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a72286397c4edc9a63eca3412be168; no voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:29.364289 24608 raft_consensus.cc:2726] T 128dc7838caa403282ed2868444be271 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:29.374097 24608 raft_consensus.cc:482] T e9f6f71aaffa41e281e67fec134a4827 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a736a8ebb164ce0a051c0fccd76c3b8)
I20240502 21:29:29.374231 24608 raft_consensus.cc:504] T e9f6f71aaffa41e281e67fec134a4827 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:29.374786 24608 leader_election.cc:288] T e9f6f71aaffa41e281e67fec134a4827 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
W20240502 21:29:29.376188 7420 leader_election.cc:334] T e9f6f71aaffa41e281e67fec134a4827 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111)
W20240502 21:29:29.376372 7420 leader_election.cc:334] T e9f6f71aaffa41e281e67fec134a4827 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:29.376475 7420 leader_election.cc:302] T e9f6f71aaffa41e281e67fec134a4827 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a72286397c4edc9a63eca3412be168; no voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:29:29.376700 24608 raft_consensus.cc:2726] T e9f6f71aaffa41e281e67fec134a4827 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:29.461720 24589 raft_consensus.cc:482] T 9d279c5db74f46519c35fe75a35589c7 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a736a8ebb164ce0a051c0fccd76c3b8)
I20240502 21:29:29.461933 24589 raft_consensus.cc:504] T 9d279c5db74f46519c35fe75a35589c7 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:29.462625 24589 leader_election.cc:288] T 9d279c5db74f46519c35fe75a35589c7 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:29:29.463110 7481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d279c5db74f46519c35fe75a35589c7" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
I20240502 21:29:29.463320 7481 raft_consensus.cc:2454] T 9d279c5db74f46519c35fe75a35589c7 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 1.
I20240502 21:29:29.463831 7281 leader_election.cc:302] T 9d279c5db74f46519c35fe75a35589c7 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, d0456c5e55a542128688904e003d403c; no voters:
W20240502 21:29:29.464149 7285 leader_election.cc:334] T 9d279c5db74f46519c35fe75a35589c7 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111)
I20240502 21:29:29.464207 24589 raft_consensus.cc:2781] T 9d279c5db74f46519c35fe75a35589c7 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:29:29.464411 24589 raft_consensus.cc:482] T 9d279c5db74f46519c35fe75a35589c7 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a736a8ebb164ce0a051c0fccd76c3b8)
I20240502 21:29:29.464527 24589 raft_consensus.cc:3037] T 9d279c5db74f46519c35fe75a35589c7 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:29.466526 24589 raft_consensus.cc:504] T 9d279c5db74f46519c35fe75a35589c7 P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:29.467186 24589 leader_election.cc:288] T 9d279c5db74f46519c35fe75a35589c7 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 election: Requested vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913)
I20240502 21:29:29.467885 7481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d279c5db74f46519c35fe75a35589c7" candidate_uuid: "d0456c5e55a542128688904e003d403c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168"
I20240502 21:29:29.468075 7481 raft_consensus.cc:3037] T 9d279c5db74f46519c35fe75a35589c7 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:29.470052 7481 raft_consensus.cc:2454] T 9d279c5db74f46519c35fe75a35589c7 P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0456c5e55a542128688904e003d403c in term 2.
I20240502 21:29:29.470384 7281 leader_election.cc:302] T 9d279c5db74f46519c35fe75a35589c7 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a72286397c4edc9a63eca3412be168, d0456c5e55a542128688904e003d403c; no voters:
W20240502 21:29:29.470687 7285 leader_election.cc:334] T 9d279c5db74f46519c35fe75a35589c7 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111)
I20240502 21:29:29.471210 24589 raft_consensus.cc:2781] T 9d279c5db74f46519c35fe75a35589c7 P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:29.471350 24589 raft_consensus.cc:686] T 9d279c5db74f46519c35fe75a35589c7 P d0456c5e55a542128688904e003d403c [term 2 LEADER]: Becoming Leader. State: Replica: d0456c5e55a542128688904e003d403c, State: Running, Role: LEADER
I20240502 21:29:29.471633 24589 consensus_queue.cc:237] T 9d279c5db74f46519c35fe75a35589c7 P d0456c5e55a542128688904e003d403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:29.475383 4299 catalog_manager.cc:5484] T 9d279c5db74f46519c35fe75a35589c7 P d0456c5e55a542128688904e003d403c reported cstate change: term changed from 1 to 2, leader changed from 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66) to d0456c5e55a542128688904e003d403c (127.29.82.70). New cstate: current_term: 2 leader_uuid: "d0456c5e55a542128688904e003d403c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } }
I20240502 21:29:29.476591 24530 raft_consensus.cc:482] T 4c06f3bc97254bff8181d3f6f02470b4 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a736a8ebb164ce0a051c0fccd76c3b8)
I20240502 21:29:29.476778 24530 raft_consensus.cc:504] T 4c06f3bc97254bff8181d3f6f02470b4 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:29:29.481982 24530 leader_election.cc:288] T 4c06f3bc97254bff8181d3f6f02470b4 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
W20240502 21:29:29.485671 7549 leader_election.cc:334] T 4c06f3bc97254bff8181d3f6f02470b4 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
W20240502 21:29:29.485967 7552 leader_election.cc:334] T 4c06f3bc97254bff8181d3f6f02470b4 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111)
I20240502 21:29:29.486107 7552 leader_election.cc:302] T 4c06f3bc97254bff8181d3f6f02470b4 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04; no voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:29:29.486641 24530 raft_consensus.cc:2726] T 4c06f3bc97254bff8181d3f6f02470b4 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:29.532888 24530 raft_consensus.cc:482] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a736a8ebb164ce0a051c0fccd76c3b8)
I20240502 21:29:29.533113 24530 raft_consensus.cc:504] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:29.533854 24530 leader_election.cc:288] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
W20240502 21:29:29.535373 7552 leader_election.cc:334] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
W20240502 21:29:29.535601 7552 leader_election.cc:334] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111)
I20240502 21:29:29.535707 7552 leader_election.cc:302] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04; no voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:29:29.535969 24530 raft_consensus.cc:2726] T 546a832255ad42bebdf8eaa455ba1c3b P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 21:29:29.574884 24530 raft_consensus.cc:482] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a736a8ebb164ce0a051c0fccd76c3b8)
I20240502 21:29:29.575095 24530 raft_consensus.cc:504] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:29.575759 24530 leader_election.cc:288] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
W20240502 21:29:29.577050 7552 leader_election.cc:334] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111)
W20240502 21:29:29.577251 7549 leader_election.cc:334] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111)
I20240502 21:29:29.577387 7549 leader_election.cc:302] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04; no voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:29:29.577642 24530 raft_consensus.cc:2726] T c9fe2e18aa6c407fbbcbb74bf45bd520 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:29.587782 24578 raft_consensus.cc:482] T 6687a533112445dc86951fc778d0725b P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:29.587965 24578 raft_consensus.cc:504] T 6687a533112445dc86951fc778d0725b P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:29:29.588608 24578 leader_election.cc:288] T 6687a533112445dc86951fc778d0725b P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
W20240502 21:29:29.590131 7281 leader_election.cc:334] T 6687a533112445dc86951fc778d0725b P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
W20240502 21:29:29.590539 7285 leader_election.cc:334] T 6687a533112445dc86951fc778d0725b P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:29.590677 7285 leader_election.cc:302] T 6687a533112445dc86951fc778d0725b P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0456c5e55a542128688904e003d403c; no voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:29.590926 24578 raft_consensus.cc:2726] T 6687a533112445dc86951fc778d0725b P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:29.593376 24589 raft_consensus.cc:482] T 171a71f8620c44909b78cb7b35cfed5e P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a736a8ebb164ce0a051c0fccd76c3b8)
I20240502 21:29:29.593547 24589 raft_consensus.cc:504] T 171a71f8620c44909b78cb7b35cfed5e P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:29:29.594157 24589 leader_election.cc:288] T 171a71f8620c44909b78cb7b35cfed5e P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:29:29.594547 24530 raft_consensus.cc:482] T 016ad8b6874c4b629882f1243ea1fba2 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a736a8ebb164ce0a051c0fccd76c3b8)
I20240502 21:29:29.594713 24530 raft_consensus.cc:504] T 016ad8b6874c4b629882f1243ea1fba2 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:29.595328 24530 leader_election.cc:288] T 016ad8b6874c4b629882f1243ea1fba2 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
W20240502 21:29:29.595693 7285 leader_election.cc:334] T 171a71f8620c44909b78cb7b35cfed5e P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111)
W20240502 21:29:29.596246 7281 leader_election.cc:334] T 171a71f8620c44909b78cb7b35cfed5e P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:29.596359 7281 leader_election.cc:302] T 171a71f8620c44909b78cb7b35cfed5e P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0456c5e55a542128688904e003d403c; no voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:29:29.596707 24589 raft_consensus.cc:2726] T 171a71f8620c44909b78cb7b35cfed5e P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:29:29.597247 7552 leader_election.cc:334] T 016ad8b6874c4b629882f1243ea1fba2 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
W20240502 21:29:29.597426 7552 leader_election.cc:334] T 016ad8b6874c4b629882f1243ea1fba2 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111)
I20240502 21:29:29.597525 7552 leader_election.cc:302] T 016ad8b6874c4b629882f1243ea1fba2 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04; no voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:29:29.597783 24530 raft_consensus.cc:2726] T 016ad8b6874c4b629882f1243ea1fba2 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:29.613502 24589 raft_consensus.cc:482] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a736a8ebb164ce0a051c0fccd76c3b8)
I20240502 21:29:29.613735 24589 raft_consensus.cc:504] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:29:29.614483 24589 leader_election.cc:288] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
W20240502 21:29:29.615826 7285 leader_election.cc:334] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111)
W20240502 21:29:29.616601 7281 leader_election.cc:334] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:29.616724 7281 leader_election.cc:302] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0456c5e55a542128688904e003d403c; no voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:29:29.617019 24589 raft_consensus.cc:2726] T 14bd28b4c6a54d98a18351b60f299a3a P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 21:29:29.619127 24578 raft_consensus.cc:482] T 8ffa252784984df1bbb0ce35a26d4865 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a736a8ebb164ce0a051c0fccd76c3b8)
I20240502 21:29:29.619144 24608 raft_consensus.cc:482] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a736a8ebb164ce0a051c0fccd76c3b8)
I20240502 21:29:29.619273 24578 raft_consensus.cc:504] T 8ffa252784984df1bbb0ce35a26d4865 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:29:29.619341 24608 raft_consensus.cc:504] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:29.619936 24578 leader_election.cc:288] T 8ffa252784984df1bbb0ce35a26d4865 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
W20240502 21:29:29.621651 7420 leader_election.cc:334] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111)
W20240502 21:29:29.621802 7285 leader_election.cc:334] T 8ffa252784984df1bbb0ce35a26d4865 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111)
I20240502 21:29:29.622077 24608 leader_election.cc:288] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
W20240502 21:29:29.622467 7416 leader_election.cc:334] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:29.622593 7416 leader_election.cc:302] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a72286397c4edc9a63eca3412be168; no voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:29:29.622962 24608 raft_consensus.cc:2726] T 2eb73e73b64b4b0f9e5116f258f2e147 P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:29:29.623697 7281 leader_election.cc:334] T 8ffa252784984df1bbb0ce35a26d4865 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111)
I20240502 21:29:29.623839 7281 leader_election.cc:302] T 8ffa252784984df1bbb0ce35a26d4865 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0456c5e55a542128688904e003d403c; no voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:29:29.624070 24578 raft_consensus.cc:2726] T 8ffa252784984df1bbb0ce35a26d4865 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:29.646919 24608 raft_consensus.cc:482] T 5c5cd4328db94367b4201a49d521a145 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a736a8ebb164ce0a051c0fccd76c3b8)
I20240502 21:29:29.647114 24608 raft_consensus.cc:504] T 5c5cd4328db94367b4201a49d521a145 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:29.647794 24608 leader_election.cc:288] T 5c5cd4328db94367b4201a49d521a145 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
W20240502 21:29:29.649075 7420 leader_election.cc:334] T 5c5cd4328db94367b4201a49d521a145 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111)
W20240502 21:29:29.649376 7416 leader_election.cc:334] T 5c5cd4328db94367b4201a49d521a145 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111)
I20240502 21:29:29.649518 7416 leader_election.cc:302] T 5c5cd4328db94367b4201a49d521a145 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a72286397c4edc9a63eca3412be168; no voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:29:29.649773 24608 raft_consensus.cc:2726] T 5c5cd4328db94367b4201a49d521a145 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:29.663995 24578 raft_consensus.cc:482] T 73413ab7b2514ff8a0f2a9a7bd580b2c P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a736a8ebb164ce0a051c0fccd76c3b8)
I20240502 21:29:29.664171 24578 raft_consensus.cc:504] T 73413ab7b2514ff8a0f2a9a7bd580b2c P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:29:29.664922 24578 leader_election.cc:288] T 73413ab7b2514ff8a0f2a9a7bd580b2c P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
W20240502 21:29:29.666046 7285 leader_election.cc:334] T 73413ab7b2514ff8a0f2a9a7bd580b2c P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111)
W20240502 21:29:29.666481 7281 leader_election.cc:334] T 73413ab7b2514ff8a0f2a9a7bd580b2c P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111)
I20240502 21:29:29.666597 7281 leader_election.cc:302] T 73413ab7b2514ff8a0f2a9a7bd580b2c P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0456c5e55a542128688904e003d403c; no voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:29:29.666849 24578 raft_consensus.cc:2726] T 73413ab7b2514ff8a0f2a9a7bd580b2c P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:29.668110 24530 raft_consensus.cc:482] T c0bc7002249e41079c92eb2364f870ab P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a736a8ebb164ce0a051c0fccd76c3b8)
I20240502 21:29:29.668355 24530 raft_consensus.cc:504] T c0bc7002249e41079c92eb2364f870ab P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:29:29.669010 24530 leader_election.cc:288] T c0bc7002249e41079c92eb2364f870ab P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
W20240502 21:29:29.670398 7552 leader_election.cc:334] T c0bc7002249e41079c92eb2364f870ab P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111)
I20240502 21:29:29.678390 24578 raft_consensus.cc:482] T 59ef00d58c244b40ba4eebe040f8ffbd P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a736a8ebb164ce0a051c0fccd76c3b8)
I20240502 21:29:29.678591 24578 raft_consensus.cc:504] T 59ef00d58c244b40ba4eebe040f8ffbd P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:29:29.679297 24578 leader_election.cc:288] T 59ef00d58c244b40ba4eebe040f8ffbd P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
W20240502 21:29:29.680815 7285 leader_election.cc:334] T 59ef00d58c244b40ba4eebe040f8ffbd P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111)
W20240502 21:29:29.680975 7281 leader_election.cc:334] T 59ef00d58c244b40ba4eebe040f8ffbd P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111)
I20240502 21:29:29.681314 7281 leader_election.cc:302] T 59ef00d58c244b40ba4eebe040f8ffbd P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0456c5e55a542128688904e003d403c; no voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:29:29.681599 24578 raft_consensus.cc:2726] T 59ef00d58c244b40ba4eebe040f8ffbd P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:29.695973 24578 raft_consensus.cc:482] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a736a8ebb164ce0a051c0fccd76c3b8)
I20240502 21:29:29.696128 24578 raft_consensus.cc:504] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:29:29.696704 24578 leader_election.cc:288] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
W20240502 21:29:29.698118 7285 leader_election.cc:334] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
W20240502 21:29:29.698334 7285 leader_election.cc:334] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111)
I20240502 21:29:29.698438 7285 leader_election.cc:302] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0456c5e55a542128688904e003d403c; no voters: 398c3c8f3195492c900439ad5e1931bb, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:29:29.698673 24578 raft_consensus.cc:2726] T 3394f5f087304842ada4212f2e524356 P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 21:29:29.724654 24150 raft_consensus.cc:482] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a736a8ebb164ce0a051c0fccd76c3b8)
I20240502 21:29:29.724866 24150 raft_consensus.cc:504] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:29.725476 24150 leader_election.cc:288] T 4dc27c0c9cee447f84e4bc1a63f64b68 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:29:29.728657 24633 raft_consensus.cc:482] T c0bc7002249e41079c92eb2364f870ab P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a736a8ebb164ce0a051c0fccd76c3b8)
I20240502 21:29:29.728824 24633 raft_consensus.cc:504] T c0bc7002249e41079c92eb2364f870ab P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:29:29.730562 7212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0bc7002249e41079c92eb2364f870ab" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39981a8dfe94462b9a73ba489af4c714" is_pre_election: true
I20240502 21:29:29.730780 24633 leader_election.cc:288] T c0bc7002249e41079c92eb2364f870ab P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:29:29.730784 7212 raft_consensus.cc:2454] T c0bc7002249e41079c92eb2364f870ab P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddd486109f5434cace13d60d633ac04 in term 1.
I20240502 21:29:29.732070 7615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0bc7002249e41079c92eb2364f870ab" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:29:29.732326 7615 raft_consensus.cc:2454] T c0bc7002249e41079c92eb2364f870ab P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 1.
I20240502 21:29:29.733242 7149 leader_election.cc:302] T c0bc7002249e41079c92eb2364f870ab P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 39981a8dfe94462b9a73ba489af4c714; no voters:
I20240502 21:29:29.733755 24633 raft_consensus.cc:2781] T c0bc7002249e41079c92eb2364f870ab P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:29:29.733856 24633 raft_consensus.cc:482] T c0bc7002249e41079c92eb2364f870ab P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a736a8ebb164ce0a051c0fccd76c3b8)
I20240502 21:29:29.733960 24633 raft_consensus.cc:3037] T c0bc7002249e41079c92eb2364f870ab P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:29.740829 24589 raft_consensus.cc:482] T d6370cfa3d804e0fa5dc2a0f71d78840 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a736a8ebb164ce0a051c0fccd76c3b8)
I20240502 21:29:29.741061 24589 raft_consensus.cc:504] T d6370cfa3d804e0fa5dc2a0f71d78840 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:29:29.741845 24589 leader_election.cc:288] T d6370cfa3d804e0fa5dc2a0f71d78840 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:29:29.742622 24633 raft_consensus.cc:504] T c0bc7002249e41079c92eb2364f870ab P 39981a8dfe94462b9a73ba489af4c714 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:29:29.742564 24578 raft_consensus.cc:482] T 5d710d1340514764b0c7d38e6bf19436 P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a736a8ebb164ce0a051c0fccd76c3b8)
I20240502 21:29:29.742955 24578 raft_consensus.cc:504] T 5d710d1340514764b0c7d38e6bf19436 P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:29:29.743788 7615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0bc7002249e41079c92eb2364f870ab" candidate_uuid: "39981a8dfe94462b9a73ba489af4c714" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:29:29.743845 24578 leader_election.cc:288] T 5d710d1340514764b0c7d38e6bf19436 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:29:29.743995 7615 raft_consensus.cc:3037] T c0bc7002249e41079c92eb2364f870ab P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:29.745419 24633 leader_election.cc:288] T c0bc7002249e41079c92eb2364f870ab P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 2 election: Requested vote from peers 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:29:29.746973 7615 raft_consensus.cc:2454] T c0bc7002249e41079c92eb2364f870ab P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39981a8dfe94462b9a73ba489af4c714 in term 2.
I20240502 21:29:29.747589 7149 leader_election.cc:302] T c0bc7002249e41079c92eb2364f870ab P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 39981a8dfe94462b9a73ba489af4c714; no voters:
W20240502 21:29:29.747902 7285 leader_election.cc:334] T d6370cfa3d804e0fa5dc2a0f71d78840 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111)
W20240502 21:29:29.748072 7285 leader_election.cc:334] T 5d710d1340514764b0c7d38e6bf19436 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111)
I20240502 21:29:29.748247 24150 raft_consensus.cc:2781] T c0bc7002249e41079c92eb2364f870ab P 39981a8dfe94462b9a73ba489af4c714 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:29.748392 24150 raft_consensus.cc:686] T c0bc7002249e41079c92eb2364f870ab P 39981a8dfe94462b9a73ba489af4c714 [term 2 LEADER]: Becoming Leader. State: Replica: 39981a8dfe94462b9a73ba489af4c714, State: Running, Role: LEADER
I20240502 21:29:29.748665 24150 consensus_queue.cc:237] T c0bc7002249e41079c92eb2364f870ab P 39981a8dfe94462b9a73ba489af4c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:29:29.751919 24603 raft_consensus.cc:482] T 3a1fadb11cd6436a94e4a67a3aad17ba P 39981a8dfe94462b9a73ba489af4c714 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a736a8ebb164ce0a051c0fccd76c3b8)
I20240502 21:29:29.752136 24603 raft_consensus.cc:504] T 3a1fadb11cd6436a94e4a67a3aad17ba P 39981a8dfe94462b9a73ba489af4c714 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:29.752918 24603 leader_election.cc:288] T 3a1fadb11cd6436a94e4a67a3aad17ba P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
I20240502 21:29:29.753576 24631 raft_consensus.cc:482] T a33fd7d5a8db41a39ba780e16a84f9b1 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a736a8ebb164ce0a051c0fccd76c3b8)
I20240502 21:29:29.753726 24631 raft_consensus.cc:504] T a33fd7d5a8db41a39ba780e16a84f9b1 P 39981a8dfe94462b9a73ba489af4c714 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:29.754289 24631 leader_election.cc:288] T a33fd7d5a8db41a39ba780e16a84f9b1 P 39981a8dfe94462b9a73ba489af4c714 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
W20240502 21:29:29.754359 7281 leader_election.cc:334] T d6370cfa3d804e0fa5dc2a0f71d78840 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111)
I20240502 21:29:29.754474 7281 leader_election.cc:302] T d6370cfa3d804e0fa5dc2a0f71d78840 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0456c5e55a542128688904e003d403c; no voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:29:29.755431 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 7129
I20240502 21:29:29.757458 24578 raft_consensus.cc:2726] T d6370cfa3d804e0fa5dc2a0f71d78840 P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:29:29.758282 7281 leader_election.cc:334] T 5d710d1340514764b0c7d38e6bf19436 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:29.758404 7281 leader_election.cc:302] T 5d710d1340514764b0c7d38e6bf19436 P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0456c5e55a542128688904e003d403c; no voters: 71e1375b799b439fa3c83ca763c039c1, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:29:29.758638 24578 raft_consensus.cc:2726] T 5d710d1340514764b0c7d38e6bf19436 P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 21:29:29.763074 7551 leader_election.cc:302] T c0bc7002249e41079c92eb2364f870ab P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 39981a8dfe94462b9a73ba489af4c714; no voters: 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:29:29.763375 24525 raft_consensus.cc:2741] T c0bc7002249e41079c92eb2364f870ab P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20240502 21:29:29.797824 24589 raft_consensus.cc:482] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 73a1f2ca1f544de9b0e3b37015f85328)
I20240502 21:29:29.798030 24589 raft_consensus.cc:504] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } }
I20240502 21:29:29.798851 24589 leader_election.cc:288] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
W20240502 21:29:29.799721 7285 consensus_peers.cc:462] T c6459a8c391249529d2672ab6c62a8af P d0456c5e55a542128688904e003d403c -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
W20240502 21:29:29.800648 7281 leader_election.cc:334] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111)
W20240502 21:29:29.800877 7281 leader_election.cc:334] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:29.801023 7281 leader_election.cc:302] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0456c5e55a542128688904e003d403c; no voters: 71e1375b799b439fa3c83ca763c039c1, 73a1f2ca1f544de9b0e3b37015f85328
I20240502 21:29:29.801007 24635 raft_consensus.cc:482] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71e1375b799b439fa3c83ca763c039c1)
I20240502 21:29:29.801187 24635 raft_consensus.cc:504] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } }
I20240502 21:29:29.801281 24589 raft_consensus.cc:2726] T 292bfad92b14456da85bdc874fb7e06e P d0456c5e55a542128688904e003d403c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 21:29:29.801927 24635 leader_election.cc:288] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
W20240502 21:29:29.803292 7416 leader_election.cc:334] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
W20240502 21:29:29.803539 7420 leader_election.cc:334] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:29.803691 7420 leader_election.cc:302] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a72286397c4edc9a63eca3412be168; no voters: 398c3c8f3195492c900439ad5e1931bb, 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:29.803956 24635 raft_consensus.cc:2726] T 81db5e77d00446cba3eb8345ee3df476 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:29:29.815835 7551 connection.cc:535] client connection to 127.29.82.69:33193 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:29:29.816059 7419 connection.cc:535] client connection to 127.29.82.69:33193 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:29:29.816455 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 7265
W20240502 21:29:29.817035 7551 consensus_peers.cc:485] T a8a3d57ff2184ab3bc09140afbffe0b8 P 0ddd486109f5434cace13d60d633ac04 -> Peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Couldn't send request to peer 39981a8dfe94462b9a73ba489af4c714. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.69:33193: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:29.817076 7419 consensus_peers.cc:485] T 402e76e6f92744fbb36653d04f861e4f P 31a72286397c4edc9a63eca3412be168 -> Peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Couldn't send request to peer 39981a8dfe94462b9a73ba489af4c714. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.69:33193: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:29.817173 7551 consensus_peers.cc:485] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 0ddd486109f5434cace13d60d633ac04 -> Peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Couldn't send request to peer 39981a8dfe94462b9a73ba489af4c714. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.69:33193: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:29.817512 24636 raft_consensus.cc:482] T 202f8b10d5674afa875da2f3929bd8da P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 73a1f2ca1f544de9b0e3b37015f85328)
I20240502 21:29:29.818079 24636 raft_consensus.cc:504] T 202f8b10d5674afa875da2f3929bd8da P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:29:29.818640 24636 leader_election.cc:288] T 202f8b10d5674afa875da2f3929bd8da P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
W20240502 21:29:29.820094 7549 leader_election.cc:334] T 202f8b10d5674afa875da2f3929bd8da P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111)
W20240502 21:29:29.820293 7549 leader_election.cc:334] T 202f8b10d5674afa875da2f3929bd8da P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:29.820405 7549 leader_election.cc:302] T 202f8b10d5674afa875da2f3929bd8da P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04; no voters: 71e1375b799b439fa3c83ca763c039c1, 73a1f2ca1f544de9b0e3b37015f85328
I20240502 21:29:29.820685 24636 raft_consensus.cc:2726] T 202f8b10d5674afa875da2f3929bd8da P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:29.823122 7389 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:29:29.867555 7419 consensus_peers.cc:485] T a85e7872c28a410496f31ecff9c71edb P 31a72286397c4edc9a63eca3412be168 -> Peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Couldn't send request to peer 39981a8dfe94462b9a73ba489af4c714. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.69:33193: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:29.883141 7551 consensus_peers.cc:485] T beaa535c1dbe48b48613582d65e751ae P 0ddd486109f5434cace13d60d633ac04 -> Peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Couldn't send request to peer 39981a8dfe94462b9a73ba489af4c714. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.69:33193: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:29.890110 7551 consensus_peers.cc:485] T 2707427ddc3d49b89c000e92e3b663d9 P 0ddd486109f5434cace13d60d633ac04 -> Peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Couldn't send request to peer 39981a8dfe94462b9a73ba489af4c714. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.69:33193: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:29.901662 7552 consensus_peers.cc:485] T 2285212d78b0460ca329949eb3140e69 P 0ddd486109f5434cace13d60d633ac04 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:29.913836 7551 consensus_peers.cc:485] T cfceb76b56924833bd7e6b3901573ee7 P 0ddd486109f5434cace13d60d633ac04 -> Peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Couldn't send request to peer 39981a8dfe94462b9a73ba489af4c714. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.69:33193: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:29.929481 7419 consensus_peers.cc:485] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 -> Peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Couldn't send request to peer 39981a8dfe94462b9a73ba489af4c714. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.69:33193: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:29.965971 7551 consensus_peers.cc:485] T 9e5434cb329e4bea85317d18f6e6867d P 0ddd486109f5434cace13d60d633ac04 -> Peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Couldn't send request to peer 39981a8dfe94462b9a73ba489af4c714. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.69:33193: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:29.967692 24525 raft_consensus.cc:482] T 648263320b474a32bfffa18b1d4a61e3 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 73a1f2ca1f544de9b0e3b37015f85328)
I20240502 21:29:29.967842 24525 raft_consensus.cc:504] T 648263320b474a32bfffa18b1d4a61e3 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:29.968410 24525 leader_election.cc:288] T 648263320b474a32bfffa18b1d4a61e3 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:29:29.968886 7481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "648263320b474a32bfffa18b1d4a61e3" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
W20240502 21:29:29.969957 7549 leader_election.cc:334] T 648263320b474a32bfffa18b1d4a61e3 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111)
I20240502 21:29:29.970086 7549 leader_election.cc:302] T 648263320b474a32bfffa18b1d4a61e3 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04; no voters: 31a72286397c4edc9a63eca3412be168, 73a1f2ca1f544de9b0e3b37015f85328
I20240502 21:29:29.970325 24525 raft_consensus.cc:2726] T 648263320b474a32bfffa18b1d4a61e3 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:30.008790 24525 raft_consensus.cc:482] T a5d47492f64f49d584246fc5007281aa P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 73a1f2ca1f544de9b0e3b37015f85328)
I20240502 21:29:30.008983 24525 raft_consensus.cc:504] T a5d47492f64f49d584246fc5007281aa P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:29:30.009652 24525 leader_election.cc:288] T a5d47492f64f49d584246fc5007281aa P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:29:30.010143 7481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d47492f64f49d584246fc5007281aa" candidate_uuid: "0ddd486109f5434cace13d60d633ac04" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "31a72286397c4edc9a63eca3412be168" is_pre_election: true
W20240502 21:29:30.011483 7549 leader_election.cc:334] T a5d47492f64f49d584246fc5007281aa P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111)
I20240502 21:29:30.011651 7549 leader_election.cc:302] T a5d47492f64f49d584246fc5007281aa P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04; no voters: 31a72286397c4edc9a63eca3412be168, 73a1f2ca1f544de9b0e3b37015f85328
I20240502 21:29:30.011924 24525 raft_consensus.cc:2726] T a5d47492f64f49d584246fc5007281aa P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:29:30.042608 7549 consensus_peers.cc:485] T 0914e3ce5a2d43788ba9d369227e4f66 P 0ddd486109f5434cace13d60d633ac04 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:29:30.061417 24635 raft_consensus.cc:482] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 73a1f2ca1f544de9b0e3b37015f85328)
I20240502 21:29:30.061606 24635 raft_consensus.cc:504] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:30.062212 24635 leader_election.cc:288] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:29:30.062728 7615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "648263320b474a32bfffa18b1d4a61e3" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04" is_pre_election: true
I20240502 21:29:30.062952 7615 raft_consensus.cc:2454] T 648263320b474a32bfffa18b1d4a61e3 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 1.
I20240502 21:29:30.063380 7418 leader_election.cc:302] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168; no voters:
I20240502 21:29:30.063660 24635 raft_consensus.cc:2781] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240502 21:29:30.063679 7416 leader_election.cc:334] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111)
I20240502 21:29:30.063792 24635 raft_consensus.cc:482] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 73a1f2ca1f544de9b0e3b37015f85328)
I20240502 21:29:30.063905 24635 raft_consensus.cc:3037] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:30.066156 24635 raft_consensus.cc:504] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:30.066754 24635 leader_election.cc:288] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 election: Requested vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 0ddd486109f5434cace13d60d633ac04 (127.29.82.72:41223)
I20240502 21:29:30.067273 7615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "648263320b474a32bfffa18b1d4a61e3" candidate_uuid: "31a72286397c4edc9a63eca3412be168" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ddd486109f5434cace13d60d633ac04"
I20240502 21:29:30.067485 7615 raft_consensus.cc:3037] T 648263320b474a32bfffa18b1d4a61e3 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:29:30.067927 7416 leader_election.cc:334] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111)
I20240502 21:29:30.069682 7615 raft_consensus.cc:2454] T 648263320b474a32bfffa18b1d4a61e3 P 0ddd486109f5434cace13d60d633ac04 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a72286397c4edc9a63eca3412be168 in term 2.
I20240502 21:29:30.070046 7418 leader_election.cc:302] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04, 31a72286397c4edc9a63eca3412be168; no voters: 73a1f2ca1f544de9b0e3b37015f85328
I20240502 21:29:30.070308 24635 raft_consensus.cc:2781] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:30.070480 24635 raft_consensus.cc:686] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168 [term 2 LEADER]: Becoming Leader. State: Replica: 31a72286397c4edc9a63eca3412be168, State: Running, Role: LEADER
I20240502 21:29:30.070731 24635 consensus_queue.cc:237] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:30.075325 4299 catalog_manager.cc:5484] T 648263320b474a32bfffa18b1d4a61e3 P 31a72286397c4edc9a63eca3412be168 reported cstate change: term changed from 1 to 2, leader changed from 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68) to 31a72286397c4edc9a63eca3412be168 (127.29.82.71). New cstate: current_term: 2 leader_uuid: "31a72286397c4edc9a63eca3412be168" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } }
W20240502 21:29:30.079337 7419 consensus_peers.cc:485] T d81b71825ba54eacb6c106bce3a838e8 P 31a72286397c4edc9a63eca3412be168 -> Peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Couldn't send request to peer 39981a8dfe94462b9a73ba489af4c714. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.69:33193: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:30.115365 7419 consensus_peers.cc:485] T 063bdcf9cea74d5cbe3143eb16fba6a5 P 31a72286397c4edc9a63eca3412be168 -> Peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Couldn't send request to peer 39981a8dfe94462b9a73ba489af4c714. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.69:33193: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:30.145112 7551 consensus_peers.cc:485] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 -> Peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Couldn't send request to peer 39981a8dfe94462b9a73ba489af4c714. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.69:33193: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:30.185914 24578 raft_consensus.cc:482] T acdc893a70224a7b82083e998910d6ef P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 73a1f2ca1f544de9b0e3b37015f85328)
I20240502 21:29:30.186148 24578 raft_consensus.cc:504] T acdc893a70224a7b82083e998910d6ef P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } }
I20240502 21:29:30.186995 24578 leader_election.cc:288] T acdc893a70224a7b82083e998910d6ef P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781)
I20240502 21:29:30.190806 24647 raft_consensus.cc:482] T dd1cf8e447e54d6e907ffa0e4681515f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 73a1f2ca1f544de9b0e3b37015f85328)
I20240502 21:29:30.190969 24647 raft_consensus.cc:504] T dd1cf8e447e54d6e907ffa0e4681515f P d0456c5e55a542128688904e003d403c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:30.191926 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 7265
I20240502 21:29:30.191967 24647 leader_election.cc:288] T dd1cf8e447e54d6e907ffa0e4681515f P d0456c5e55a542128688904e003d403c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:29:30.193241 7481 raft_consensus.cc:1261] T 9d279c5db74f46519c35fe75a35589c7 P 31a72286397c4edc9a63eca3412be168 [term 2 FOLLOWER]: Refusing update from remote peer d0456c5e55a542128688904e003d403c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240502 21:29:30.196539 7420 consensus_peers.cc:485] T 0f822150741f4ec1a3a9472572160359 P 31a72286397c4edc9a63eca3412be168 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:30.196650 7552 consensus_peers.cc:485] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:29:30.224990 24636 raft_consensus.cc:482] T 796c7f3cc1f9426daecc89add3c6aa57 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a736a8ebb164ce0a051c0fccd76c3b8)
I20240502 21:29:30.225242 24636 raft_consensus.cc:504] T 796c7f3cc1f9426daecc89add3c6aa57 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:29:30.226110 24636 leader_election.cc:288] T 796c7f3cc1f9426daecc89add3c6aa57 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:29:30.227349 24636 raft_consensus.cc:482] T 104c676bff964b8b8566b2200ce238c7 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 73a1f2ca1f544de9b0e3b37015f85328)
I20240502 21:29:30.227514 24636 raft_consensus.cc:504] T 104c676bff964b8b8566b2200ce238c7 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
W20240502 21:29:30.227674 7549 leader_election.cc:334] T 796c7f3cc1f9426daecc89add3c6aa57 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111)
I20240502 21:29:30.228060 24636 leader_election.cc:288] T 104c676bff964b8b8566b2200ce238c7 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537)
W20240502 21:29:30.228225 7552 leader_election.cc:334] T 796c7f3cc1f9426daecc89add3c6aa57 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111)
I20240502 21:29:30.228327 7552 leader_election.cc:302] T 796c7f3cc1f9426daecc89add3c6aa57 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04; no voters: 73a1f2ca1f544de9b0e3b37015f85328, 7a736a8ebb164ce0a051c0fccd76c3b8
I20240502 21:29:30.228758 24636 raft_consensus.cc:2726] T 796c7f3cc1f9426daecc89add3c6aa57 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:29:30.229617 7549 leader_election.cc:334] T 104c676bff964b8b8566b2200ce238c7 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449): Network error: Client connection negotiation failed: client connection to 127.29.82.68:42449: connect: Connection refused (error 111)
W20240502 21:29:30.230044 7552 leader_election.cc:334] T 104c676bff964b8b8566b2200ce238c7 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111)
I20240502 21:29:30.230137 7552 leader_election.cc:302] T 104c676bff964b8b8566b2200ce238c7 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04; no voters: 398c3c8f3195492c900439ad5e1931bb, 73a1f2ca1f544de9b0e3b37015f85328
I20240502 21:29:30.230402 24636 raft_consensus.cc:2726] T 104c676bff964b8b8566b2200ce238c7 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:29:30.251528 7420 connection.cc:535] server connection from 127.29.82.70:39533 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:29:30.251952 7419 connection.cc:535] client connection to 127.29.82.70:40719 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:29:30.252064 7549 connection.cc:535] server connection from 127.29.82.70:37961 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:29:30.252312 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 7399
W20240502 21:29:30.253243 7551 consensus_peers.cc:485] T 616eda3cb218475c8bf0108cc6b90810 P 0ddd486109f5434cace13d60d633ac04 -> Peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Couldn't send request to peer d0456c5e55a542128688904e003d403c. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.70:40719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:30.253381 7551 consensus_peers.cc:485] T 2ae670e3d8ae49f2b8c7df65b45b179e P 0ddd486109f5434cace13d60d633ac04 -> Peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Couldn't send request to peer d0456c5e55a542128688904e003d403c. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.70:40719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:30.253480 7551 consensus_peers.cc:485] T 6065bdba6e2840b8b368d8f92d1fa260 P 0ddd486109f5434cace13d60d633ac04 -> Peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Couldn't send request to peer d0456c5e55a542128688904e003d403c. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.70:40719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:30.253520 7419 consensus_peers.cc:485] T adbccf5386624b28af36ff893ac9378f P 31a72286397c4edc9a63eca3412be168 -> Peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Couldn't send request to peer d0456c5e55a542128688904e003d403c. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.70:40719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:30.253563 7551 consensus_peers.cc:485] T 8991236419e642bfa438b9e45a1b8235 P 0ddd486109f5434cace13d60d633ac04 -> Peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Couldn't send request to peer d0456c5e55a542128688904e003d403c. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.70:40719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:30.253647 7419 consensus_peers.cc:485] T 269eb0e719804925961e4bcd17c18596 P 31a72286397c4edc9a63eca3412be168 -> Peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Couldn't send request to peer d0456c5e55a542128688904e003d403c. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.70:40719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:30.253711 7419 consensus_peers.cc:485] T 39260f3652b9473687ce1abb9c39223e P 31a72286397c4edc9a63eca3412be168 -> Peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Couldn't send request to peer d0456c5e55a542128688904e003d403c. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.70:40719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:30.253787 7419 consensus_peers.cc:485] T 80304ab48274408393c2e245477c3a78 P 31a72286397c4edc9a63eca3412be168 -> Peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Couldn't send request to peer d0456c5e55a542128688904e003d403c. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.70:40719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:30.258230 7524 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:29:30.261092 7551 consensus_peers.cc:485] T 1bddb5e40d464de082c1d586618c86d7 P 0ddd486109f5434cace13d60d633ac04 -> Peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Couldn't send request to peer d0456c5e55a542128688904e003d403c. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.70:40719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:30.400228 7551 consensus_peers.cc:485] T ca332df7aaef4ddd9fd3266e8003b55f P 0ddd486109f5434cace13d60d633ac04 -> Peer d0456c5e55a542128688904e003d403c (127.29.82.70:40719): Couldn't send request to peer d0456c5e55a542128688904e003d403c. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.70:40719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:30.537815 7549 consensus_peers.cc:485] T a7a0167853fa420586455b82fb893c46 P 0ddd486109f5434cace13d60d633ac04 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:30.552819 7552 consensus_peers.cc:485] T 38ce97f9f9ac4398a16958931ef7262f P 0ddd486109f5434cace13d60d633ac04 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:29:30.616437 24635 raft_consensus.cc:482] T 686a74b8c1034112b293c4c7c6bcd99a P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 73a1f2ca1f544de9b0e3b37015f85328)
I20240502 21:29:30.618649 24635 raft_consensus.cc:504] T 686a74b8c1034112b293c4c7c6bcd99a P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:29:30.619544 24635 leader_election.cc:288] T 686a74b8c1034112b293c4c7c6bcd99a P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:29:30.620237 24538 raft_consensus.cc:482] T be423032bc4b459b8e8e63a2379e37ac P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39981a8dfe94462b9a73ba489af4c714)
I20240502 21:29:30.620409 24635 raft_consensus.cc:482] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 398c3c8f3195492c900439ad5e1931bb)
I20240502 21:29:30.620596 24635 raft_consensus.cc:504] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "398c3c8f3195492c900439ad5e1931bb" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 34537 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } }
I20240502 21:29:30.621376 24635 leader_election.cc:288] T fd610e340cd3409397d8f23b623b4074 P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537), 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781)
I20240502 21:29:30.620412 24538 raft_consensus.cc:504] T be423032bc4b459b8e8e63a2379e37ac P 31a72286397c4edc9a63eca3412be168 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } }
I20240502 21:29:30.622545 24538 leader_election.cc:288] T be423032bc4b459b8e8e63a2379e37ac P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0456c5e55a542128688904e003d403c (127.29.82.70:40719), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
W20240502 21:29:30.623425 7416 consensus_peers.cc:485] T fb2cc722452f4d9798353d1bfae4bd4f P 31a72286397c4edc9a63eca3412be168 -> Peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Couldn't send request to peer 71e1375b799b439fa3c83ca763c039c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:30.623668 7416 leader_election.cc:334] T 686a74b8c1034112b293c4c7c6bcd99a P 31a72286397c4edc9a63eca3412be168 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:30.623745 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 7399
W20240502 21:29:30.680505 7550 connection.cc:535] server connection from 127.29.82.71:43299 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:29:30.681474 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 7533
I20240502 21:29:30.682080 24525 raft_consensus.cc:482] T 9e6feb60ae03404e85ef76499de190c3 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39981a8dfe94462b9a73ba489af4c714)
I20240502 21:29:30.682256 24525 raft_consensus.cc:504] T 9e6feb60ae03404e85ef76499de190c3 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "71e1375b799b439fa3c83ca763c039c1" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 42781 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:30.682931 24525 leader_election.cc:288] T 9e6feb60ae03404e85ef76499de190c3 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
W20240502 21:29:30.683066 7549 consensus_peers.cc:485] T d558bcaf20bd46548936d5fb946d7710 P 0ddd486109f5434cace13d60d633ac04 -> Peer 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913): Couldn't send request to peer 31a72286397c4edc9a63eca3412be168. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.71:33913: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:30.684092 7551 leader_election.cc:334] T 9e6feb60ae03404e85ef76499de190c3 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193): Network error: Client connection negotiation failed: client connection to 127.29.82.69:33193: connect: Connection refused (error 111)
W20240502 21:29:30.685158 7549 leader_election.cc:334] T 9e6feb60ae03404e85ef76499de190c3 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71e1375b799b439fa3c83ca763c039c1 (127.29.82.65:42781): Network error: Client connection negotiation failed: client connection to 127.29.82.65:42781: connect: Connection refused (error 111)
I20240502 21:29:30.685297 7549 leader_election.cc:302] T 9e6feb60ae03404e85ef76499de190c3 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ddd486109f5434cace13d60d633ac04; no voters: 39981a8dfe94462b9a73ba489af4c714, 71e1375b799b439fa3c83ca763c039c1
I20240502 21:29:30.685601 24525 raft_consensus.cc:2726] T 9e6feb60ae03404e85ef76499de190c3 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:29:30.688514 7657 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:29:31.033718 24525 raft_consensus.cc:482] T 86fa4255c6554ba28e141de3cb328976 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 73a1f2ca1f544de9b0e3b37015f85328)
I20240502 21:29:31.036711 24525 raft_consensus.cc:504] T 86fa4255c6554ba28e141de3cb328976 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a736a8ebb164ce0a051c0fccd76c3b8" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 35781 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } }
I20240502 21:29:31.038061 24525 leader_election.cc:288] T 86fa4255c6554ba28e141de3cb328976 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193)
I20240502 21:29:31.039549 24665 raft_consensus.cc:482] T 85fac458a4204f40a85d4ecba4718f58 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 73a1f2ca1f544de9b0e3b37015f85328)
I20240502 21:29:31.041491 24662 raft_consensus.cc:482] T 4cc82d5106804490993007f9f75943aa P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39981a8dfe94462b9a73ba489af4c714)
I20240502 21:29:31.041648 24662 raft_consensus.cc:504] T 4cc82d5106804490993007f9f75943aa P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } } peers { permanent_uuid: "39981a8dfe94462b9a73ba489af4c714" member_type: VOTER last_known_addr { host: "127.29.82.69" port: 33193 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } }
I20240502 21:29:31.042275 24662 leader_election.cc:288] T 4cc82d5106804490993007f9f75943aa P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39981a8dfe94462b9a73ba489af4c714 (127.29.82.69:33193), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449)
I20240502 21:29:31.042562 24665 raft_consensus.cc:504] T 85fac458a4204f40a85d4ecba4718f58 P 0ddd486109f5434cace13d60d633ac04 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a72286397c4edc9a63eca3412be168" member_type: VOTER last_known_addr { host: "127.29.82.71" port: 33913 } } peers { permanent_uuid: "73a1f2ca1f544de9b0e3b37015f85328" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 42449 } } peers { permanent_uuid: "d0456c5e55a542128688904e003d403c" member_type: VOTER last_known_addr { host: "127.29.82.70" port: 40719 } } peers { permanent_uuid: "0ddd486109f5434cace13d60d633ac04" member_type: VOTER last_known_addr { host: "127.29.82.72" port: 41223 } }
I20240502 21:29:31.043731 24665 leader_election.cc:288] T 85fac458a4204f40a85d4ecba4718f58 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31a72286397c4edc9a63eca3412be168 (127.29.82.71:33913), 73a1f2ca1f544de9b0e3b37015f85328 (127.29.82.68:42449), d0456c5e55a542128688904e003d403c (127.29.82.70:40719)
W20240502 21:29:31.045360 7552 consensus_peers.cc:485] T 6d0334be884c45c3bd9e49095454dfb1 P 0ddd486109f5434cace13d60d633ac04 -> Peer 398c3c8f3195492c900439ad5e1931bb (127.29.82.67:34537): Couldn't send request to peer 398c3c8f3195492c900439ad5e1931bb. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.67:34537: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:29:31.045648 7552 consensus_peers.cc:485] T e8f42cdee5c24db982392d2057b588a2 P 0ddd486109f5434cace13d60d633ac04 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:29:31.045791 7552 leader_election.cc:334] T 86fa4255c6554ba28e141de3cb328976 P 0ddd486109f5434cace13d60d633ac04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111)
W20240502 21:29:31.045961 7552 consensus_peers.cc:485] T fd95f653fd5f4a038e4bc51dbd72e8e6 P 0ddd486109f5434cace13d60d633ac04 -> Peer 7a736a8ebb164ce0a051c0fccd76c3b8 (127.29.82.66:35781): Couldn't send request to peer 7a736a8ebb164ce0a051c0fccd76c3b8. Status: Network error: Client connection negotiation failed: client connection to 127.29.82.66:35781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:29:31.046011 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 7533
I20240502 21:29:31.103025 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 4269
I20240502 21:29:31.109387 4331 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:29:31.438120 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 4269
2024-05-02T21:29:31Z chronyd exiting
I20240502 21:29:31.511982 30025 test_util.cc:175] -----------------------------------------------
I20240502 21:29:31.512068 30025 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714685124710458-30025-0
[ FAILED ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/0, where GetParam() = 4-byte object <00-00 00-00> (191227 ms)
[----------] 1 test from DDLDuringRebalancingTest (191227 ms total)
[----------] 1 test from RebalancingDuringElectionStormTest
[ RUN ] RebalancingDuringElectionStormTest.RoundRobin/0
I20240502 21:29:31.514297 30025 test_util.cc:268] Using random seed: 1282964881
I20240502 21:29:31.516260 30025 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:29:31.516361 30025 ts_itest-base.cc:113] --------------
I20240502 21:29:31.516414 30025 ts_itest-base.cc:114] 8 tablet servers
I20240502 21:29:31.516485 30025 ts_itest-base.cc:115] 3 replicas per TS
I20240502 21:29:31.516536 30025 ts_itest-base.cc:116] --------------
2024-05-02T21:29:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:29:31Z Disabled control of system clock
I20240502 21:29:31.550442 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.82.126:45283
--webserver_interface=127.29.82.126
--webserver_port=0
--builtin_ntp_servers=127.29.82.84:41041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.82.126:45283
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240502 21:29:31.892355 24680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:31.892699 24680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:29:31.892853 24680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:31.900344 24680 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:29:31.900461 24680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 21:29:31.900543 24680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:29:31.900638 24680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 21:29:31.900707 24680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 21:29:31.909862 24680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:41041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.29.82.126:45283
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.82.126:45283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.29.82.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:29:31.911229 24680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:29:31.913121 24680 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:29:31.924826 24685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:31.925544 24686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:31.926230 24688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:29:31.926914 24680 server_base.cc:1034] running on GCE node
I20240502 21:29:31.927724 24680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:29:31.929122 24680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:29:31.930332 24680 hybrid_clock.cc:648] HybridClock initialized: now 1714685371930264 us; error 54 us; skew 500 ppm
I20240502 21:29:31.930716 24680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:29:31.936287 24680 webserver.cc:460] Webserver started at http://127.29.82.126:37629/ using document root <none> and password file <none>
I20240502 21:29:31.937000 24680 fs_manager.cc:359] Metadata directory not provided
I20240502 21:29:31.937114 24680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:29:31.937443 24680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:29:31.940094 24680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "612802580f944293b9f6c35843609603"
format_stamp: "Formatted at 2024-05-02 21:29:31 on dist-test-slave-xzsm"
I20240502 21:29:31.940891 24680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "612802580f944293b9f6c35843609603"
format_stamp: "Formatted at 2024-05-02 21:29:31 on dist-test-slave-xzsm"
I20240502 21:29:31.945940 24680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 21:29:31.949223 24694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:31.950500 24680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 21:29:31.950687 24680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal
uuid: "612802580f944293b9f6c35843609603"
format_stamp: "Formatted at 2024-05-02 21:29:31 on dist-test-slave-xzsm"
I20240502 21:29:31.950873 24680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:29:31.988874 24680 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:29:31.989866 24680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:29:31.990154 24680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:29:32.008744 24680 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.126:45283
I20240502 21:29:32.008764 24747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.126:45283 every 8 connection(s)
I20240502 21:29:32.010186 24680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 21:29:32.014050 24748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:32.020539 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 24680
I20240502 21:29:32.020779 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 21:29:32.024111 24748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 612802580f944293b9f6c35843609603: Bootstrap starting.
I20240502 21:29:32.028374 24748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 612802580f944293b9f6c35843609603: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:32.029989 24748 log.cc:824] T 00000000000000000000000000000000 P 612802580f944293b9f6c35843609603: Log is configured to *not* fsync() on all Append() calls
I20240502 21:29:32.033073 24748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 612802580f944293b9f6c35843609603: No bootstrap required, opened a new log
I20240502 21:29:32.040262 24748 raft_consensus.cc:348] T 00000000000000000000000000000000 P 612802580f944293b9f6c35843609603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "612802580f944293b9f6c35843609603" member_type: VOTER last_known_addr { host: "127.29.82.126" port: 45283 } }
I20240502 21:29:32.040583 24748 raft_consensus.cc:374] T 00000000000000000000000000000000 P 612802580f944293b9f6c35843609603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:32.040665 24748 raft_consensus.cc:729] T 00000000000000000000000000000000 P 612802580f944293b9f6c35843609603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 612802580f944293b9f6c35843609603, State: Initialized, Role: FOLLOWER
I20240502 21:29:32.041208 24748 consensus_queue.cc:260] T 00000000000000000000000000000000 P 612802580f944293b9f6c35843609603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "612802580f944293b9f6c35843609603" member_type: VOTER last_known_addr { host: "127.29.82.126" port: 45283 } }
I20240502 21:29:32.041405 24748 raft_consensus.cc:388] T 00000000000000000000000000000000 P 612802580f944293b9f6c35843609603 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:29:32.041497 24748 raft_consensus.cc:482] T 00000000000000000000000000000000 P 612802580f944293b9f6c35843609603 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:29:32.041618 24748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 612802580f944293b9f6c35843609603 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:32.043962 24748 raft_consensus.cc:504] T 00000000000000000000000000000000 P 612802580f944293b9f6c35843609603 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "612802580f944293b9f6c35843609603" member_type: VOTER last_known_addr { host: "127.29.82.126" port: 45283 } }
I20240502 21:29:32.044404 24748 leader_election.cc:302] T 00000000000000000000000000000000 P 612802580f944293b9f6c35843609603 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 612802580f944293b9f6c35843609603; no voters:
I20240502 21:29:32.044927 24748 leader_election.cc:288] T 00000000000000000000000000000000 P 612802580f944293b9f6c35843609603 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:29:32.045082 24753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 612802580f944293b9f6c35843609603 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:32.045754 24753 raft_consensus.cc:686] T 00000000000000000000000000000000 P 612802580f944293b9f6c35843609603 [term 1 LEADER]: Becoming Leader. State: Replica: 612802580f944293b9f6c35843609603, State: Running, Role: LEADER
I20240502 21:29:32.046245 24753 consensus_queue.cc:237] T 00000000000000000000000000000000 P 612802580f944293b9f6c35843609603 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "612802580f944293b9f6c35843609603" member_type: VOTER last_known_addr { host: "127.29.82.126" port: 45283 } }
I20240502 21:29:32.046988 24748 sys_catalog.cc:564] T 00000000000000000000000000000000 P 612802580f944293b9f6c35843609603 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:29:32.049082 24755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 612802580f944293b9f6c35843609603 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 612802580f944293b9f6c35843609603. Latest consensus state: current_term: 1 leader_uuid: "612802580f944293b9f6c35843609603" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "612802580f944293b9f6c35843609603" member_type: VOTER last_known_addr { host: "127.29.82.126" port: 45283 } } }
I20240502 21:29:32.049508 24755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 612802580f944293b9f6c35843609603 [sys.catalog]: This master's current role is: LEADER
I20240502 21:29:32.050354 24754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 612802580f944293b9f6c35843609603 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "612802580f944293b9f6c35843609603" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "612802580f944293b9f6c35843609603" member_type: VOTER last_known_addr { host: "127.29.82.126" port: 45283 } } }
I20240502 21:29:32.050693 24754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 612802580f944293b9f6c35843609603 [sys.catalog]: This master's current role is: LEADER
I20240502 21:29:32.054814 24760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:29:32.060429 24760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:29:32.067773 24760 catalog_manager.cc:1347] Generated new cluster ID: a265393815604d3ba9e11fbb956cd1c3
I20240502 21:29:32.067884 24760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:29:32.074791 24760 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:29:32.076479 24760 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:29:32.094446 24760 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 612802580f944293b9f6c35843609603: Generated new TSK 0
I20240502 21:29:32.095525 24760 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:29:32.103894 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.65:0
--local_ip_for_outbound_sockets=127.29.82.65
--webserver_interface=127.29.82.65
--webserver_port=0
--tserver_master_addrs=127.29.82.126:45283
--builtin_ntp_servers=127.29.82.84:41041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 21:29:32.448144 24772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:32.448417 24772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:29:32.448544 24772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:32.456228 24772 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:29:32.456342 24772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:29:32.456509 24772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.65
I20240502 21:29:32.465337 24772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:41041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.29.82.65
--webserver_port=0
--tserver_master_addrs=127.29.82.126:45283
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.65
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:29:32.466570 24772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:29:32.468365 24772 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:29:32.481086 24777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:32.481596 24780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:32.482115 24778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:29:32.483129 24772 server_base.cc:1034] running on GCE node
I20240502 21:29:32.483794 24772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:29:32.484753 24772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:29:32.485952 24772 hybrid_clock.cc:648] HybridClock initialized: now 1714685372485904 us; error 40 us; skew 500 ppm
I20240502 21:29:32.486269 24772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:29:32.488505 24772 webserver.cc:460] Webserver started at http://127.29.82.65:33101/ using document root <none> and password file <none>
I20240502 21:29:32.489240 24772 fs_manager.cc:359] Metadata directory not provided
I20240502 21:29:32.489338 24772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:29:32.489660 24772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:29:32.492233 24772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a171ad73f86643afb1444248c0d0bc8b"
format_stamp: "Formatted at 2024-05-02 21:29:32 on dist-test-slave-xzsm"
I20240502 21:29:32.492951 24772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a171ad73f86643afb1444248c0d0bc8b"
format_stamp: "Formatted at 2024-05-02 21:29:32 on dist-test-slave-xzsm"
I20240502 21:29:32.497900 24772 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 21:29:32.501205 24786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:32.502449 24772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 21:29:32.502588 24772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a171ad73f86643afb1444248c0d0bc8b"
format_stamp: "Formatted at 2024-05-02 21:29:32 on dist-test-slave-xzsm"
I20240502 21:29:32.502774 24772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:29:32.518523 24772 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:29:32.519341 24772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:29:32.519639 24772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:29:32.520581 24772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:29:32.522190 24772 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:29:32.522276 24772 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:32.522357 24772 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:29:32.522411 24772 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:32.554803 24772 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.65:33021
I20240502 21:29:32.554841 24903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.65:33021 every 8 connection(s)
I20240502 21:29:32.556197 24772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 21:29:32.557888 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 24772
I20240502 21:29:32.558059 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 21:29:32.561635 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.66:0
--local_ip_for_outbound_sockets=127.29.82.66
--webserver_interface=127.29.82.66
--webserver_port=0
--tserver_master_addrs=127.29.82.126:45283
--builtin_ntp_servers=127.29.82.84:41041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:29:32.569528 24904 heartbeater.cc:347] Connected to a master server at 127.29.82.126:45283
I20240502 21:29:32.569813 24904 heartbeater.cc:464] Registering TS with master...
I20240502 21:29:32.570508 24904 heartbeater.cc:511] Master 127.29.82.126:45283 requested a full tablet report, sending...
I20240502 21:29:32.572539 24712 ts_manager.cc:194] Registered new tserver with Master: a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:32.574064 24712 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.65:54461
W20240502 21:29:32.895275 24908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:32.895560 24908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:29:32.895687 24908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:32.903131 24908 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:29:32.903252 24908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:29:32.903415 24908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.66
I20240502 21:29:32.912744 24908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:41041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.29.82.66
--webserver_port=0
--tserver_master_addrs=127.29.82.126:45283
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.66
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:29:32.914127 24908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:29:32.916081 24908 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:29:32.929042 24917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:32.929737 24915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:32.930305 24914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:29:32.930928 24908 server_base.cc:1034] running on GCE node
I20240502 21:29:32.931525 24908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:29:32.932515 24908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:29:32.933722 24908 hybrid_clock.cc:648] HybridClock initialized: now 1714685372933665 us; error 46 us; skew 500 ppm
I20240502 21:29:32.934037 24908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:29:32.936324 24908 webserver.cc:460] Webserver started at http://127.29.82.66:36519/ using document root <none> and password file <none>
I20240502 21:29:32.937076 24908 fs_manager.cc:359] Metadata directory not provided
I20240502 21:29:32.937196 24908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:29:32.937558 24908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:29:32.940210 24908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "64d97dd1674d43ccadd7b4a38eeff557"
format_stamp: "Formatted at 2024-05-02 21:29:32 on dist-test-slave-xzsm"
I20240502 21:29:32.940990 24908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "64d97dd1674d43ccadd7b4a38eeff557"
format_stamp: "Formatted at 2024-05-02 21:29:32 on dist-test-slave-xzsm"
I20240502 21:29:32.946218 24908 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 21:29:32.949644 24923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:32.950902 24908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20240502 21:29:32.951068 24908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "64d97dd1674d43ccadd7b4a38eeff557"
format_stamp: "Formatted at 2024-05-02 21:29:32 on dist-test-slave-xzsm"
I20240502 21:29:32.951289 24908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:29:32.973452 24908 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:29:32.974234 24908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:29:32.974505 24908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:29:32.975432 24908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:29:32.976967 24908 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:29:32.977072 24908 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:32.977192 24908 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:29:32.977275 24908 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:33.010546 24908 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.66:39721
I20240502 21:29:33.010579 25037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.66:39721 every 8 connection(s)
I20240502 21:29:33.011919 24908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 21:29:33.016526 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 24908
I20240502 21:29:33.016726 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 21:29:33.020498 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.67:0
--local_ip_for_outbound_sockets=127.29.82.67
--webserver_interface=127.29.82.67
--webserver_port=0
--tserver_master_addrs=127.29.82.126:45283
--builtin_ntp_servers=127.29.82.84:41041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:29:33.025599 25038 heartbeater.cc:347] Connected to a master server at 127.29.82.126:45283
I20240502 21:29:33.025905 25038 heartbeater.cc:464] Registering TS with master...
I20240502 21:29:33.026661 25038 heartbeater.cc:511] Master 127.29.82.126:45283 requested a full tablet report, sending...
I20240502 21:29:33.028059 24712 ts_manager.cc:194] Registered new tserver with Master: 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:33.028856 24712 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.66:55063
W20240502 21:29:33.360594 25041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:33.360882 25041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:29:33.361023 25041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:33.368376 25041 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:29:33.368510 25041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:29:33.368675 25041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.67
I20240502 21:29:33.377467 25041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:41041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.29.82.67
--webserver_port=0
--tserver_master_addrs=127.29.82.126:45283
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.67
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:29:33.378829 25041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:29:33.380659 25041 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:29:33.393509 25046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:33.394009 25049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:33.394522 25047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:29:33.395431 25041 server_base.cc:1034] running on GCE node
I20240502 21:29:33.396028 25041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:29:33.396890 25041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:29:33.398074 25041 hybrid_clock.cc:648] HybridClock initialized: now 1714685373398034 us; error 34 us; skew 500 ppm
I20240502 21:29:33.398375 25041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:29:33.400559 25041 webserver.cc:460] Webserver started at http://127.29.82.67:36623/ using document root <none> and password file <none>
I20240502 21:29:33.401273 25041 fs_manager.cc:359] Metadata directory not provided
I20240502 21:29:33.401379 25041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:29:33.401719 25041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:29:33.404274 25041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "5bf90e514781418da921c046aa5d9ced"
format_stamp: "Formatted at 2024-05-02 21:29:33 on dist-test-slave-xzsm"
I20240502 21:29:33.405014 25041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "5bf90e514781418da921c046aa5d9ced"
format_stamp: "Formatted at 2024-05-02 21:29:33 on dist-test-slave-xzsm"
I20240502 21:29:33.410125 25041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20240502 21:29:33.413617 25055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:33.414978 25041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240502 21:29:33.415138 25041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "5bf90e514781418da921c046aa5d9ced"
format_stamp: "Formatted at 2024-05-02 21:29:33 on dist-test-slave-xzsm"
I20240502 21:29:33.415333 25041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:29:33.442725 25041 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:29:33.443539 25041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:29:33.443845 25041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:29:33.444810 25041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:29:33.446280 25041 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:29:33.446369 25041 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:33.446485 25041 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:29:33.446560 25041 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:33.479753 25041 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.67:37875
I20240502 21:29:33.479820 25168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.67:37875 every 8 connection(s)
I20240502 21:29:33.481105 25041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:29:33.487210 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 25041
I20240502 21:29:33.487412 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 21:29:33.490944 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.68:0
--local_ip_for_outbound_sockets=127.29.82.68
--webserver_interface=127.29.82.68
--webserver_port=0
--tserver_master_addrs=127.29.82.126:45283
--builtin_ntp_servers=127.29.82.84:41041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:29:33.494701 25169 heartbeater.cc:347] Connected to a master server at 127.29.82.126:45283
I20240502 21:29:33.494998 25169 heartbeater.cc:464] Registering TS with master...
I20240502 21:29:33.495868 25169 heartbeater.cc:511] Master 127.29.82.126:45283 requested a full tablet report, sending...
I20240502 21:29:33.497195 24712 ts_manager.cc:194] Registered new tserver with Master: 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:33.497978 24712 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.67:47757
I20240502 21:29:33.576829 24904 heartbeater.cc:503] Master 127.29.82.126:45283 was elected leader, sending a full tablet report...
W20240502 21:29:33.841943 25174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:33.842218 25174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:29:33.842308 25174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:33.849655 25174 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:29:33.849767 25174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:29:33.849889 25174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.68
I20240502 21:29:33.858784 25174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:41041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.29.82.68
--webserver_port=0
--tserver_master_addrs=127.29.82.126:45283
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.68
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:29:33.860062 25174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:29:33.861771 25174 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:29:33.873924 25179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:33.874416 25180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:33.874889 25182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:29:33.875813 25174 server_base.cc:1034] running on GCE node
I20240502 21:29:33.876359 25174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:29:33.877205 25174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:29:33.878417 25174 hybrid_clock.cc:648] HybridClock initialized: now 1714685373878366 us; error 34 us; skew 500 ppm
I20240502 21:29:33.878759 25174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:29:33.880978 25174 webserver.cc:460] Webserver started at http://127.29.82.68:43431/ using document root <none> and password file <none>
I20240502 21:29:33.881685 25174 fs_manager.cc:359] Metadata directory not provided
I20240502 21:29:33.881780 25174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:29:33.882061 25174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:29:33.884738 25174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "45b7d5b9696b4046b522c880c748fe6c"
format_stamp: "Formatted at 2024-05-02 21:29:33 on dist-test-slave-xzsm"
I20240502 21:29:33.885453 25174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "45b7d5b9696b4046b522c880c748fe6c"
format_stamp: "Formatted at 2024-05-02 21:29:33 on dist-test-slave-xzsm"
I20240502 21:29:33.890594 25174 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 21:29:33.893885 25188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:33.895179 25174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240502 21:29:33.895313 25174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "45b7d5b9696b4046b522c880c748fe6c"
format_stamp: "Formatted at 2024-05-02 21:29:33 on dist-test-slave-xzsm"
I20240502 21:29:33.895499 25174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:29:33.925858 25174 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:29:33.926654 25174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:29:33.926920 25174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:29:33.927881 25174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:29:33.929283 25174 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:29:33.929368 25174 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:33.929487 25174 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:29:33.929543 25174 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:33.960031 25174 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.68:43099
I20240502 21:29:33.960114 25306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.68:43099 every 8 connection(s)
I20240502 21:29:33.961293 25174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:29:33.965611 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 25174
I20240502 21:29:33.965781 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 21:29:33.969240 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.69:0
--local_ip_for_outbound_sockets=127.29.82.69
--webserver_interface=127.29.82.69
--webserver_port=0
--tserver_master_addrs=127.29.82.126:45283
--builtin_ntp_servers=127.29.82.84:41041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:29:33.974217 25308 heartbeater.cc:347] Connected to a master server at 127.29.82.126:45283
I20240502 21:29:33.974565 25308 heartbeater.cc:464] Registering TS with master...
I20240502 21:29:33.975526 25308 heartbeater.cc:511] Master 127.29.82.126:45283 requested a full tablet report, sending...
I20240502 21:29:33.977124 24712 ts_manager.cc:194] Registered new tserver with Master: 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:33.977924 24712 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.68:47669
I20240502 21:29:34.031726 25038 heartbeater.cc:503] Master 127.29.82.126:45283 was elected leader, sending a full tablet report...
W20240502 21:29:34.306649 25311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:34.306988 25311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:29:34.307152 25311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:34.314684 25311 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:29:34.314810 25311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:29:34.314981 25311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.69
I20240502 21:29:34.324326 25311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:41041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.29.82.69
--webserver_port=0
--tserver_master_addrs=127.29.82.126:45283
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.69
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:29:34.325687 25311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:29:34.327512 25311 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:29:34.340209 25316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:34.340737 25319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:34.341238 25317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:29:34.342051 25311 server_base.cc:1034] running on GCE node
I20240502 21:29:34.342581 25311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:29:34.343401 25311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:29:34.344586 25311 hybrid_clock.cc:648] HybridClock initialized: now 1714685374344535 us; error 41 us; skew 500 ppm
I20240502 21:29:34.344866 25311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:29:34.346830 25311 webserver.cc:460] Webserver started at http://127.29.82.69:38963/ using document root <none> and password file <none>
I20240502 21:29:34.347481 25311 fs_manager.cc:359] Metadata directory not provided
I20240502 21:29:34.347612 25311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:29:34.347942 25311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:29:34.350360 25311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "0b56aba0e3e9491c9f5e9ee2b281e316"
format_stamp: "Formatted at 2024-05-02 21:29:34 on dist-test-slave-xzsm"
I20240502 21:29:34.351112 25311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "0b56aba0e3e9491c9f5e9ee2b281e316"
format_stamp: "Formatted at 2024-05-02 21:29:34 on dist-test-slave-xzsm"
I20240502 21:29:34.355844 25311 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.003s
I20240502 21:29:34.359565 25325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:34.360811 25311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:29:34.360982 25311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "0b56aba0e3e9491c9f5e9ee2b281e316"
format_stamp: "Formatted at 2024-05-02 21:29:34 on dist-test-slave-xzsm"
I20240502 21:29:34.361156 25311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:29:34.382153 25311 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:29:34.382941 25311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:29:34.383216 25311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:29:34.384160 25311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:29:34.385599 25311 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:29:34.385699 25311 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:34.385818 25311 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:29:34.385900 25311 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:34.420763 25311 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.69:46087
I20240502 21:29:34.420841 25442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.69:46087 every 8 connection(s)
I20240502 21:29:34.422041 25311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:29:34.424072 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 25311
I20240502 21:29:34.424275 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 21:29:34.427891 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.70:0
--local_ip_for_outbound_sockets=127.29.82.70
--webserver_interface=127.29.82.70
--webserver_port=0
--tserver_master_addrs=127.29.82.126:45283
--builtin_ntp_servers=127.29.82.84:41041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:29:34.435424 25443 heartbeater.cc:347] Connected to a master server at 127.29.82.126:45283
I20240502 21:29:34.435794 25443 heartbeater.cc:464] Registering TS with master...
I20240502 21:29:34.436736 25443 heartbeater.cc:511] Master 127.29.82.126:45283 requested a full tablet report, sending...
I20240502 21:29:34.438198 24712 ts_manager.cc:194] Registered new tserver with Master: 0b56aba0e3e9491c9f5e9ee2b281e316 (127.29.82.69:46087)
I20240502 21:29:34.438946 24712 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.69:52055
I20240502 21:29:34.500474 25169 heartbeater.cc:503] Master 127.29.82.126:45283 was elected leader, sending a full tablet report...
W20240502 21:29:34.763095 25446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:34.763361 25446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:29:34.763446 25446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:34.770720 25446 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:29:34.770841 25446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:29:34.770957 25446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.70
I20240502 21:29:34.779827 25446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:41041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.29.82.70
--webserver_port=0
--tserver_master_addrs=127.29.82.126:45283
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.70
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:29:34.781137 25446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:29:34.782934 25446 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:29:34.795197 25451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:34.795778 25452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:34.796269 25454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:29:34.797169 25446 server_base.cc:1034] running on GCE node
I20240502 21:29:34.797753 25446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:29:34.798660 25446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:29:34.799882 25446 hybrid_clock.cc:648] HybridClock initialized: now 1714685374799830 us; error 27 us; skew 500 ppm
I20240502 21:29:34.800202 25446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:29:34.802366 25446 webserver.cc:460] Webserver started at http://127.29.82.70:43687/ using document root <none> and password file <none>
I20240502 21:29:34.803069 25446 fs_manager.cc:359] Metadata directory not provided
I20240502 21:29:34.803189 25446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:29:34.803519 25446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:29:34.806018 25446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "506baf9b2c0a44c28c3e7fd08733f795"
format_stamp: "Formatted at 2024-05-02 21:29:34 on dist-test-slave-xzsm"
I20240502 21:29:34.806818 25446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "506baf9b2c0a44c28c3e7fd08733f795"
format_stamp: "Formatted at 2024-05-02 21:29:34 on dist-test-slave-xzsm"
I20240502 21:29:34.811867 25446 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240502 21:29:34.815163 25460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:34.816510 25446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:29:34.816674 25446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "506baf9b2c0a44c28c3e7fd08733f795"
format_stamp: "Formatted at 2024-05-02 21:29:34 on dist-test-slave-xzsm"
I20240502 21:29:34.816859 25446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:29:34.841037 25446 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:29:34.841830 25446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:29:34.842087 25446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:29:34.843004 25446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:29:34.844460 25446 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:29:34.844561 25446 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:34.844686 25446 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:29:34.844765 25446 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:34.876336 25446 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.70:36971
I20240502 21:29:34.876384 25575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.70:36971 every 8 connection(s)
I20240502 21:29:34.877709 25446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 21:29:34.883203 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 25446
I20240502 21:29:34.883397 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 21:29:34.886827 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.71:0
--local_ip_for_outbound_sockets=127.29.82.71
--webserver_interface=127.29.82.71
--webserver_port=0
--tserver_master_addrs=127.29.82.126:45283
--builtin_ntp_servers=127.29.82.84:41041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:29:34.890585 25576 heartbeater.cc:347] Connected to a master server at 127.29.82.126:45283
I20240502 21:29:34.890879 25576 heartbeater.cc:464] Registering TS with master...
I20240502 21:29:34.891650 25576 heartbeater.cc:511] Master 127.29.82.126:45283 requested a full tablet report, sending...
I20240502 21:29:34.893043 24712 ts_manager.cc:194] Registered new tserver with Master: 506baf9b2c0a44c28c3e7fd08733f795 (127.29.82.70:36971)
I20240502 21:29:34.894155 24712 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.70:43203
I20240502 21:29:34.980482 25308 heartbeater.cc:503] Master 127.29.82.126:45283 was elected leader, sending a full tablet report...
W20240502 21:29:35.221916 25579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:35.222254 25579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:29:35.222359 25579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:35.229730 25579 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:29:35.229835 25579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:29:35.229960 25579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.71
I20240502 21:29:35.238608 25579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:41041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.29.82.71
--webserver_port=0
--tserver_master_addrs=127.29.82.126:45283
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.71
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:29:35.239876 25579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:29:35.241565 25579 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:29:35.253639 25584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:35.254384 25585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:35.254977 25587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:29:35.255699 25579 server_base.cc:1034] running on GCE node
I20240502 21:29:35.256263 25579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:29:35.257083 25579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:29:35.258296 25579 hybrid_clock.cc:648] HybridClock initialized: now 1714685375258255 us; error 33 us; skew 500 ppm
I20240502 21:29:35.258592 25579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:29:35.260784 25579 webserver.cc:460] Webserver started at http://127.29.82.71:40287/ using document root <none> and password file <none>
I20240502 21:29:35.261514 25579 fs_manager.cc:359] Metadata directory not provided
I20240502 21:29:35.261627 25579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:29:35.261950 25579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:29:35.264549 25579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "332de10467624bdb8cdcb69669543330"
format_stamp: "Formatted at 2024-05-02 21:29:35 on dist-test-slave-xzsm"
I20240502 21:29:35.265347 25579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "332de10467624bdb8cdcb69669543330"
format_stamp: "Formatted at 2024-05-02 21:29:35 on dist-test-slave-xzsm"
I20240502 21:29:35.270238 25579 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 21:29:35.273409 25593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:35.274590 25579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 21:29:35.274737 25579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "332de10467624bdb8cdcb69669543330"
format_stamp: "Formatted at 2024-05-02 21:29:35 on dist-test-slave-xzsm"
I20240502 21:29:35.274909 25579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:29:35.310950 25579 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:29:35.311880 25579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:29:35.312153 25579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:29:35.313119 25579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:29:35.314687 25579 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:29:35.314790 25579 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:35.314910 25579 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:29:35.314991 25579 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:35.348047 25579 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.71:36399
I20240502 21:29:35.348142 25709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.71:36399 every 8 connection(s)
I20240502 21:29:35.349341 25579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 21:29:35.352324 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 25579
I20240502 21:29:35.352522 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 21:29:35.356043 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.72:0
--local_ip_for_outbound_sockets=127.29.82.72
--webserver_interface=127.29.82.72
--webserver_port=0
--tserver_master_addrs=127.29.82.126:45283
--builtin_ntp_servers=127.29.82.84:41041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:29:35.370368 25710 heartbeater.cc:347] Connected to a master server at 127.29.82.126:45283
I20240502 21:29:35.370704 25710 heartbeater.cc:464] Registering TS with master...
I20240502 21:29:35.371641 25710 heartbeater.cc:511] Master 127.29.82.126:45283 requested a full tablet report, sending...
I20240502 21:29:35.373148 24712 ts_manager.cc:194] Registered new tserver with Master: 332de10467624bdb8cdcb69669543330 (127.29.82.71:36399)
I20240502 21:29:35.373905 24712 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.71:56477
I20240502 21:29:35.441771 25443 heartbeater.cc:503] Master 127.29.82.126:45283 was elected leader, sending a full tablet report...
W20240502 21:29:35.694795 25713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:35.695101 25713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:29:35.695255 25713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:35.702693 25713 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:29:35.702831 25713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:29:35.703006 25713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.72
I20240502 21:29:35.712116 25713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:41041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.29.82.72
--webserver_port=0
--tserver_master_addrs=127.29.82.126:45283
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.72
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:29:35.713452 25713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:29:35.715335 25713 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:29:35.727815 25720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:35.728557 25719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:35.729092 25722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:29:35.729673 25713 server_base.cc:1034] running on GCE node
I20240502 21:29:35.730212 25713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:29:35.731050 25713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:29:35.732256 25713 hybrid_clock.cc:648] HybridClock initialized: now 1714685375732213 us; error 34 us; skew 500 ppm
I20240502 21:29:35.732564 25713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:29:35.734722 25713 webserver.cc:460] Webserver started at http://127.29.82.72:35125/ using document root <none> and password file <none>
I20240502 21:29:35.735420 25713 fs_manager.cc:359] Metadata directory not provided
I20240502 21:29:35.735518 25713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:29:35.735865 25713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:29:35.738474 25713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "eb2ceee828af41b8bb7f9bf11fc2c573"
format_stamp: "Formatted at 2024-05-02 21:29:35 on dist-test-slave-xzsm"
I20240502 21:29:35.739225 25713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "eb2ceee828af41b8bb7f9bf11fc2c573"
format_stamp: "Formatted at 2024-05-02 21:29:35 on dist-test-slave-xzsm"
I20240502 21:29:35.744519 25713 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240502 21:29:35.747869 25728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:35.749095 25713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 21:29:35.749240 25713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "eb2ceee828af41b8bb7f9bf11fc2c573"
format_stamp: "Formatted at 2024-05-02 21:29:35 on dist-test-slave-xzsm"
I20240502 21:29:35.749491 25713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:29:35.772691 25713 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:29:35.773583 25713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:29:35.773855 25713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:29:35.774799 25713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:29:35.776468 25713 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:29:35.776571 25713 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:35.776692 25713 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:29:35.776768 25713 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:35.808955 25713 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.72:44277
I20240502 21:29:35.809003 25844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.72:44277 every 8 connection(s)
I20240502 21:29:35.810290 25713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 21:29:35.815738 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 25713
I20240502 21:29:35.815941 30025 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 21:29:35.823927 25845 heartbeater.cc:347] Connected to a master server at 127.29.82.126:45283
I20240502 21:29:35.824180 25845 heartbeater.cc:464] Registering TS with master...
I20240502 21:29:35.824867 25845 heartbeater.cc:511] Master 127.29.82.126:45283 requested a full tablet report, sending...
I20240502 21:29:35.826117 24712 ts_manager.cc:194] Registered new tserver with Master: eb2ceee828af41b8bb7f9bf11fc2c573 (127.29.82.72:44277)
I20240502 21:29:35.826905 24712 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.72:50987
I20240502 21:29:35.832144 30025 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 21:29:35.847899 30025 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 21:29:35.848006 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 25311
I20240502 21:29:35.856300 25437 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:29:35.896994 25576 heartbeater.cc:503] Master 127.29.82.126:45283 was elected leader, sending a full tablet report...
I20240502 21:29:35.936045 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 25311
I20240502 21:29:35.953742 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 25446
I20240502 21:29:35.962039 25570 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:29:36.044394 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 25446
I20240502 21:29:36.062732 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 25579
I20240502 21:29:36.071943 25705 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:29:36.152324 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 25579
I20240502 21:29:36.169641 30025 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu pid 25713
I20240502 21:29:36.177868 25840 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:29:36.258688 30025 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu with pid 25713
I20240502 21:29:40.028180 24712 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53668:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:29:40.079852 25240 tablet_service.cc:1467] Processing CreateTablet for tablet a851d0bede1444c8a96ea45ea948bd59 (DEFAULT_TABLE table=rebalance_test_table_0 [id=080ee320f67e46e99b4c0148bc595360]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:40.081121 25241 tablet_service.cc:1467] Processing CreateTablet for tablet b4874706759a403992f9662c84a62b91 (DEFAULT_TABLE table=rebalance_test_table_0 [id=080ee320f67e46e99b4c0148bc595360]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:40.081555 25240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a851d0bede1444c8a96ea45ea948bd59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:40.084405 25241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4874706759a403992f9662c84a62b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:40.085220 25104 tablet_service.cc:1467] Processing CreateTablet for tablet b4874706759a403992f9662c84a62b91 (DEFAULT_TABLE table=rebalance_test_table_0 [id=080ee320f67e46e99b4c0148bc595360]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:40.086723 25104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4874706759a403992f9662c84a62b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:40.087724 24837 tablet_service.cc:1467] Processing CreateTablet for tablet b4874706759a403992f9662c84a62b91 (DEFAULT_TABLE table=rebalance_test_table_0 [id=080ee320f67e46e99b4c0148bc595360]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:40.088150 24836 tablet_service.cc:1467] Processing CreateTablet for tablet 4e3d3eab014a466fa4a421f7d9d20fba (DEFAULT_TABLE table=rebalance_test_table_0 [id=080ee320f67e46e99b4c0148bc595360]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:40.089075 24835 tablet_service.cc:1467] Processing CreateTablet for tablet a851d0bede1444c8a96ea45ea948bd59 (DEFAULT_TABLE table=rebalance_test_table_0 [id=080ee320f67e46e99b4c0148bc595360]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:40.089239 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3d3eab014a466fa4a421f7d9d20fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:40.089838 24835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a851d0bede1444c8a96ea45ea948bd59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:40.090368 24971 tablet_service.cc:1467] Processing CreateTablet for tablet 4e3d3eab014a466fa4a421f7d9d20fba (DEFAULT_TABLE table=rebalance_test_table_0 [id=080ee320f67e46e99b4c0148bc595360]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:40.091444 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4874706759a403992f9662c84a62b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:40.091853 24971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3d3eab014a466fa4a421f7d9d20fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:40.096603 25103 tablet_service.cc:1467] Processing CreateTablet for tablet 4e3d3eab014a466fa4a421f7d9d20fba (DEFAULT_TABLE table=rebalance_test_table_0 [id=080ee320f67e46e99b4c0148bc595360]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:40.097095 25103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3d3eab014a466fa4a421f7d9d20fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:40.107435 24970 tablet_service.cc:1467] Processing CreateTablet for tablet a851d0bede1444c8a96ea45ea948bd59 (DEFAULT_TABLE table=rebalance_test_table_0 [id=080ee320f67e46e99b4c0148bc595360]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:40.108153 24970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a851d0bede1444c8a96ea45ea948bd59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:40.128756 25874 tablet_bootstrap.cc:492] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c: Bootstrap starting.
I20240502 21:29:40.132774 25876 tablet_bootstrap.cc:492] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b: Bootstrap starting.
I20240502 21:29:40.133983 25874 tablet_bootstrap.cc:654] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:40.135677 25874 log.cc:824] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c: Log is configured to *not* fsync() on all Append() calls
I20240502 21:29:40.141103 25877 tablet_bootstrap.cc:492] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557: Bootstrap starting.
I20240502 21:29:40.142503 25876 tablet_bootstrap.cc:654] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:40.144011 25876 log.cc:824] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b: Log is configured to *not* fsync() on all Append() calls
I20240502 21:29:40.144156 25875 tablet_bootstrap.cc:492] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced: Bootstrap starting.
I20240502 21:29:40.144970 25877 tablet_bootstrap.cc:654] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:40.146183 25877 log.cc:824] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557: Log is configured to *not* fsync() on all Append() calls
I20240502 21:29:40.146652 25874 tablet_bootstrap.cc:492] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c: No bootstrap required, opened a new log
I20240502 21:29:40.147070 25874 ts_tablet_manager.cc:1367] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.006s
I20240502 21:29:40.148440 25875 tablet_bootstrap.cc:654] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:40.149956 25875 log.cc:824] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced: Log is configured to *not* fsync() on all Append() calls
I20240502 21:29:40.153209 25877 tablet_bootstrap.cc:492] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557: No bootstrap required, opened a new log
I20240502 21:29:40.153498 25877 ts_tablet_manager.cc:1367] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20240502 21:29:40.157356 25874 raft_consensus.cc:348] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:40.157858 25874 raft_consensus.cc:374] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:40.157974 25874 raft_consensus.cc:729] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Initialized, Role: FOLLOWER
I20240502 21:29:40.158427 25876 tablet_bootstrap.cc:492] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b: No bootstrap required, opened a new log
I20240502 21:29:40.158720 25876 ts_tablet_manager.cc:1367] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b: Time spent bootstrapping tablet: real 0.026s user 0.015s sys 0.003s
I20240502 21:29:40.158692 25874 consensus_queue.cc:260] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:40.160202 25875 tablet_bootstrap.cc:492] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced: No bootstrap required, opened a new log
I20240502 21:29:40.160526 25875 ts_tablet_manager.cc:1367] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.003s
I20240502 21:29:40.162917 25877 raft_consensus.cc:348] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:40.163357 25877 raft_consensus.cc:374] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:40.163473 25877 raft_consensus.cc:729] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Initialized, Role: FOLLOWER
I20240502 21:29:40.164191 25877 consensus_queue.cc:260] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:40.171274 25875 raft_consensus.cc:348] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:40.172081 25875 raft_consensus.cc:374] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:40.172235 25875 raft_consensus.cc:729] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Initialized, Role: FOLLOWER
I20240502 21:29:40.172487 25877 ts_tablet_manager.cc:1398] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557: Time spent starting tablet: real 0.019s user 0.008s sys 0.010s
I20240502 21:29:40.173053 25877 tablet_bootstrap.cc:492] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557: Bootstrap starting.
I20240502 21:29:40.173204 25875 consensus_queue.cc:260] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:40.175302 25877 tablet_bootstrap.cc:654] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:40.171551 25876 raft_consensus.cc:348] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:40.176918 25876 raft_consensus.cc:374] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:40.177026 25876 raft_consensus.cc:729] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Initialized, Role: FOLLOWER
I20240502 21:29:40.179044 25874 ts_tablet_manager.cc:1398] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c: Time spent starting tablet: real 0.032s user 0.011s sys 0.017s
I20240502 21:29:40.179531 25874 tablet_bootstrap.cc:492] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c: Bootstrap starting.
I20240502 21:29:40.179886 25876 consensus_queue.cc:260] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:40.181310 25877 tablet_bootstrap.cc:492] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557: No bootstrap required, opened a new log
I20240502 21:29:40.181464 25877 ts_tablet_manager.cc:1367] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240502 21:29:40.181643 25874 tablet_bootstrap.cc:654] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:40.183301 25875 ts_tablet_manager.cc:1398] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced: Time spent starting tablet: real 0.023s user 0.011s sys 0.005s
I20240502 21:29:40.183848 25875 tablet_bootstrap.cc:492] T b4874706759a403992f9662c84a62b91 P 5bf90e514781418da921c046aa5d9ced: Bootstrap starting.
I20240502 21:29:40.185683 25875 tablet_bootstrap.cc:654] T b4874706759a403992f9662c84a62b91 P 5bf90e514781418da921c046aa5d9ced: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:40.186688 25882 raft_consensus.cc:482] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:40.186992 25882 raft_consensus.cc:504] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:40.187747 25877 raft_consensus.cc:348] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:40.188050 25877 raft_consensus.cc:374] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:40.188162 25877 raft_consensus.cc:729] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Initialized, Role: FOLLOWER
I20240502 21:29:40.188562 25877 consensus_queue.cc:260] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:40.189502 25877 ts_tablet_manager.cc:1398] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240502 21:29:40.189508 25882 leader_election.cc:288] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:40.191192 25875 tablet_bootstrap.cc:492] T b4874706759a403992f9662c84a62b91 P 5bf90e514781418da921c046aa5d9ced: No bootstrap required, opened a new log
I20240502 21:29:40.191365 25875 ts_tablet_manager.cc:1367] T b4874706759a403992f9662c84a62b91 P 5bf90e514781418da921c046aa5d9ced: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20240502 21:29:40.192770 25875 raft_consensus.cc:348] T b4874706759a403992f9662c84a62b91 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:40.193065 25875 raft_consensus.cc:374] T b4874706759a403992f9662c84a62b91 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:40.193169 25875 raft_consensus.cc:729] T b4874706759a403992f9662c84a62b91 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Initialized, Role: FOLLOWER
I20240502 21:29:40.193445 25875 consensus_queue.cc:260] T b4874706759a403992f9662c84a62b91 P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:40.195251 25875 ts_tablet_manager.cc:1398] T b4874706759a403992f9662c84a62b91 P 5bf90e514781418da921c046aa5d9ced: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:29:40.198478 25876 ts_tablet_manager.cc:1398] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b: Time spent starting tablet: real 0.040s user 0.019s sys 0.003s
I20240502 21:29:40.199031 25876 tablet_bootstrap.cc:492] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b: Bootstrap starting.
I20240502 21:29:40.201707 25876 tablet_bootstrap.cc:654] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:40.204685 25874 tablet_bootstrap.cc:492] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c: No bootstrap required, opened a new log
I20240502 21:29:40.205032 25874 ts_tablet_manager.cc:1367] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.015s
I20240502 21:29:40.206549 25874 raft_consensus.cc:348] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:40.206866 25874 raft_consensus.cc:374] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:40.206992 25874 raft_consensus.cc:729] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Initialized, Role: FOLLOWER
I20240502 21:29:40.207242 25874 consensus_queue.cc:260] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:40.207444 25876 tablet_bootstrap.cc:492] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b: No bootstrap required, opened a new log
I20240502 21:29:40.207605 25876 ts_tablet_manager.cc:1367] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20240502 21:29:40.208000 25874 ts_tablet_manager.cc:1398] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:29:40.208781 25876 raft_consensus.cc:348] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:40.209003 25876 raft_consensus.cc:374] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:40.209837 25876 raft_consensus.cc:729] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Initialized, Role: FOLLOWER
I20240502 21:29:40.210346 25876 consensus_queue.cc:260] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:40.210551 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4874706759a403992f9662c84a62b91" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b" is_pre_election: true
I20240502 21:29:40.210851 25876 ts_tablet_manager.cc:1398] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:29:40.211163 25876 tablet_bootstrap.cc:492] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b: Bootstrap starting.
I20240502 21:29:40.211169 24857 raft_consensus.cc:2454] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 0.
I20240502 21:29:40.212492 25191 leader_election.cc:302] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:40.212641 25876 tablet_bootstrap.cc:654] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:40.212952 25882 raft_consensus.cc:2781] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:40.213125 25882 raft_consensus.cc:482] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:40.213265 25882 raft_consensus.cc:3037] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:40.216085 25876 tablet_bootstrap.cc:492] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b: No bootstrap required, opened a new log
I20240502 21:29:40.216252 25876 ts_tablet_manager.cc:1367] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 21:29:40.216543 25882 raft_consensus.cc:504] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:40.217356 25882 leader_election.cc:288] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
W20240502 21:29:40.217798 25309 tablet.cc:2369] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:29:40.217870 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4874706759a403992f9662c84a62b91" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:40.218084 24857 raft_consensus.cc:3037] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:40.218199 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4874706759a403992f9662c84a62b91" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:40.218258 25124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4874706759a403992f9662c84a62b91" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced" is_pre_election: true
I20240502 21:29:40.218824 25124 raft_consensus.cc:2454] T b4874706759a403992f9662c84a62b91 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 0.
I20240502 21:29:40.217149 25876 raft_consensus.cc:348] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:40.219305 25876 raft_consensus.cc:374] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:40.219395 25876 raft_consensus.cc:729] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Initialized, Role: FOLLOWER
I20240502 21:29:40.219707 25876 consensus_queue.cc:260] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:40.220352 25876 ts_tablet_manager.cc:1398] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:29:40.221119 24857 raft_consensus.cc:2454] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 1.
I20240502 21:29:40.221514 25191 leader_election.cc:302] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters: 5bf90e514781418da921c046aa5d9ced
I20240502 21:29:40.221849 25882 raft_consensus.cc:2781] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:40.222299 25882 raft_consensus.cc:686] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:40.222815 25882 consensus_queue.cc:237] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:40.226931 24712 catalog_manager.cc:5484] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 1 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
W20240502 21:29:40.237982 25171 tablet.cc:2369] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 21:29:40.269006 25039 tablet.cc:2369] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 21:29:40.313320 24905 tablet.cc:2369] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:29:40.335531 25882 raft_consensus.cc:482] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:40.335846 25882 raft_consensus.cc:504] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:40.336689 25882 leader_election.cc:288] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:40.337713 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851d0bede1444c8a96ea45ea948bd59" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b" is_pre_election: true
I20240502 21:29:40.338003 24857 raft_consensus.cc:2454] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 0.
I20240502 21:29:40.338562 25191 leader_election.cc:302] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:40.338850 25882 raft_consensus.cc:2781] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:40.338945 25882 raft_consensus.cc:482] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:40.339051 25882 raft_consensus.cc:3037] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:40.341969 25882 raft_consensus.cc:504] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:40.342514 25882 leader_election.cc:288] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:40.343065 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851d0bede1444c8a96ea45ea948bd59" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:40.343293 24857 raft_consensus.cc:3037] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:40.344276 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851d0bede1444c8a96ea45ea948bd59" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557" is_pre_election: true
I20240502 21:29:40.344337 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851d0bede1444c8a96ea45ea948bd59" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:40.344825 24991 raft_consensus.cc:3037] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:40.345974 24857 raft_consensus.cc:2454] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 1.
I20240502 21:29:40.346343 25191 leader_election.cc:302] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:40.346585 25882 raft_consensus.cc:2781] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:40.346804 25882 raft_consensus.cc:686] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:40.347074 25882 consensus_queue.cc:237] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:40.347620 24991 raft_consensus.cc:2454] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 1.
I20240502 21:29:40.349956 24712 catalog_manager.cc:5484] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 1 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:40.443964 25886 raft_consensus.cc:482] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:40.444164 25886 raft_consensus.cc:504] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:40.445818 25886 leader_election.cc:288] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:40.452747 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3d3eab014a466fa4a421f7d9d20fba" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced" is_pre_election: true
I20240502 21:29:40.452900 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3d3eab014a466fa4a421f7d9d20fba" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557" is_pre_election: true
I20240502 21:29:40.453007 25123 raft_consensus.cc:2454] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 0.
I20240502 21:29:40.453171 24991 raft_consensus.cc:2454] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 0.
I20240502 21:29:40.453487 24789 leader_election.cc:302] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:40.453853 25886 raft_consensus.cc:2781] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:40.453976 25886 raft_consensus.cc:482] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:40.454071 25886 raft_consensus.cc:3037] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:40.456259 25886 raft_consensus.cc:504] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:40.456815 25886 leader_election.cc:288] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 1 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:40.457211 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3d3eab014a466fa4a421f7d9d20fba" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:40.457284 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3d3eab014a466fa4a421f7d9d20fba" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:40.457414 25123 raft_consensus.cc:3037] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:40.457487 24991 raft_consensus.cc:3037] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:40.459241 24991 raft_consensus.cc:2454] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 1.
I20240502 21:29:40.459527 25123 raft_consensus.cc:2454] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 1.
I20240502 21:29:40.459625 24789 leader_election.cc:302] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:40.460050 25886 raft_consensus.cc:2781] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:40.460448 25886 raft_consensus.cc:686] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [term 1 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:40.460897 25886 consensus_queue.cc:237] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:40.465154 24712 catalog_manager.cc:5484] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 0 to 1, leader changed from <none> to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 1 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:40.624079 25123 raft_consensus.cc:3037] T b4874706759a403992f9662c84a62b91 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:40.627205 25882 consensus_queue.cc:1035] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:40.636405 25894 consensus_queue.cc:1035] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:40.732537 25894 consensus_queue.cc:1035] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:40.738266 25882 consensus_queue.cc:1035] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:29:40.869977 25901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:40.870342 25901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:40.877691 25901 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 21:29:40.919025 25886 consensus_queue.cc:1035] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:40.925503 25886 consensus_queue.cc:1035] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 272.443 ms
time per row: 0.136221 ms
I20240502 21:29:41.300244 24710 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53668:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:29:41.309787 24971 tablet_service.cc:1467] Processing CreateTablet for tablet 52153a02d80043f0a1840b0e5aadd9b3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=1dc026f0b57949d09ef2d4f923652d54]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:41.309839 25103 tablet_service.cc:1467] Processing CreateTablet for tablet 52153a02d80043f0a1840b0e5aadd9b3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=1dc026f0b57949d09ef2d4f923652d54]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:41.309931 25241 tablet_service.cc:1467] Processing CreateTablet for tablet 52153a02d80043f0a1840b0e5aadd9b3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=1dc026f0b57949d09ef2d4f923652d54]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:41.310304 24971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52153a02d80043f0a1840b0e5aadd9b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:41.310353 25103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52153a02d80043f0a1840b0e5aadd9b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:41.310366 25241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52153a02d80043f0a1840b0e5aadd9b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:41.311885 25104 tablet_service.cc:1467] Processing CreateTablet for tablet 69afb121cb564de3a9bec5e3f1cc0492 (DEFAULT_TABLE table=rebalance_test_table_1 [id=1dc026f0b57949d09ef2d4f923652d54]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:41.312278 25104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69afb121cb564de3a9bec5e3f1cc0492. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:41.314990 24837 tablet_service.cc:1467] Processing CreateTablet for tablet a3a378f7a44648c3accc9dcba847040d (DEFAULT_TABLE table=rebalance_test_table_1 [id=1dc026f0b57949d09ef2d4f923652d54]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:41.315479 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3a378f7a44648c3accc9dcba847040d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:41.316076 25102 tablet_service.cc:1467] Processing CreateTablet for tablet a3a378f7a44648c3accc9dcba847040d (DEFAULT_TABLE table=rebalance_test_table_1 [id=1dc026f0b57949d09ef2d4f923652d54]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:41.316486 25102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3a378f7a44648c3accc9dcba847040d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:41.320390 24969 tablet_service.cc:1467] Processing CreateTablet for tablet a3a378f7a44648c3accc9dcba847040d (DEFAULT_TABLE table=rebalance_test_table_1 [id=1dc026f0b57949d09ef2d4f923652d54]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:41.320474 24970 tablet_service.cc:1467] Processing CreateTablet for tablet 69afb121cb564de3a9bec5e3f1cc0492 (DEFAULT_TABLE table=rebalance_test_table_1 [id=1dc026f0b57949d09ef2d4f923652d54]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:41.320889 24969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3a378f7a44648c3accc9dcba847040d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:41.322502 25240 tablet_service.cc:1467] Processing CreateTablet for tablet 69afb121cb564de3a9bec5e3f1cc0492 (DEFAULT_TABLE table=rebalance_test_table_1 [id=1dc026f0b57949d09ef2d4f923652d54]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:41.323047 25240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69afb121cb564de3a9bec5e3f1cc0492. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:41.328354 24970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69afb121cb564de3a9bec5e3f1cc0492. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:41.333580 25967 tablet_bootstrap.cc:492] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557: Bootstrap starting.
I20240502 21:29:41.336369 25967 tablet_bootstrap.cc:654] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:41.336855 25970 tablet_bootstrap.cc:492] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b: Bootstrap starting.
I20240502 21:29:41.339211 25970 tablet_bootstrap.cc:654] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:41.344411 25966 tablet_bootstrap.cc:492] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced: Bootstrap starting.
I20240502 21:29:41.344664 25967 tablet_bootstrap.cc:492] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557: No bootstrap required, opened a new log
I20240502 21:29:41.344815 25967 ts_tablet_manager.cc:1367] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240502 21:29:41.345747 25967 raft_consensus.cc:348] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:41.345961 25967 raft_consensus.cc:374] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:41.346088 25967 raft_consensus.cc:729] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Initialized, Role: FOLLOWER
I20240502 21:29:41.346357 25967 consensus_queue.cc:260] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:41.346575 25966 tablet_bootstrap.cc:654] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:41.347162 25967 ts_tablet_manager.cc:1398] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:29:41.347831 25967 tablet_bootstrap.cc:492] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557: Bootstrap starting.
I20240502 21:29:41.350608 25967 tablet_bootstrap.cc:654] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:41.353592 25969 tablet_bootstrap.cc:492] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c: Bootstrap starting.
I20240502 21:29:41.354847 25967 tablet_bootstrap.cc:492] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557: No bootstrap required, opened a new log
I20240502 21:29:41.355217 25967 ts_tablet_manager.cc:1367] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240502 21:29:41.355520 25969 tablet_bootstrap.cc:654] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:41.356079 25967 raft_consensus.cc:348] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:41.356216 25966 tablet_bootstrap.cc:492] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced: No bootstrap required, opened a new log
I20240502 21:29:41.356308 25967 raft_consensus.cc:374] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:41.356401 25967 raft_consensus.cc:729] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Initialized, Role: FOLLOWER
I20240502 21:29:41.356437 25966 ts_tablet_manager.cc:1367] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.001s
I20240502 21:29:41.356768 25967 consensus_queue.cc:260] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:41.357201 25970 tablet_bootstrap.cc:492] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b: No bootstrap required, opened a new log
I20240502 21:29:41.357353 25970 ts_tablet_manager.cc:1367] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.008s
I20240502 21:29:41.357667 25967 ts_tablet_manager.cc:1398] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:29:41.358007 25967 tablet_bootstrap.cc:492] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557: Bootstrap starting.
I20240502 21:29:41.358098 25966 raft_consensus.cc:348] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:41.358297 25966 raft_consensus.cc:374] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:41.358218 25970 raft_consensus.cc:348] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:41.358376 25966 raft_consensus.cc:729] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Initialized, Role: FOLLOWER
I20240502 21:29:41.358414 25970 raft_consensus.cc:374] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:41.358520 25970 raft_consensus.cc:729] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Initialized, Role: FOLLOWER
I20240502 21:29:41.358682 25966 consensus_queue.cc:260] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:41.360222 25967 tablet_bootstrap.cc:654] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:41.361513 25966 ts_tablet_manager.cc:1398] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 21:29:41.363452 25970 consensus_queue.cc:260] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:41.363734 25966 tablet_bootstrap.cc:492] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced: Bootstrap starting.
I20240502 21:29:41.364228 25970 ts_tablet_manager.cc:1398] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 21:29:41.366650 25966 tablet_bootstrap.cc:654] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:41.367195 25969 tablet_bootstrap.cc:492] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c: No bootstrap required, opened a new log
I20240502 21:29:41.367357 25969 ts_tablet_manager.cc:1367] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.009s
I20240502 21:29:41.367842 25967 tablet_bootstrap.cc:492] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557: No bootstrap required, opened a new log
I20240502 21:29:41.368048 25967 ts_tablet_manager.cc:1367] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20240502 21:29:41.368286 25969 raft_consensus.cc:348] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:41.368502 25969 raft_consensus.cc:374] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:41.368603 25969 raft_consensus.cc:729] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Initialized, Role: FOLLOWER
I20240502 21:29:41.368868 25967 raft_consensus.cc:348] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:41.368875 25969 consensus_queue.cc:260] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:41.369068 25967 raft_consensus.cc:374] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:41.369168 25967 raft_consensus.cc:729] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Initialized, Role: FOLLOWER
I20240502 21:29:41.369495 25969 ts_tablet_manager.cc:1398] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:29:41.369417 25967 consensus_queue.cc:260] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:41.369963 25969 tablet_bootstrap.cc:492] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c: Bootstrap starting.
I20240502 21:29:41.369989 25967 ts_tablet_manager.cc:1398] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:29:41.371328 25966 tablet_bootstrap.cc:492] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced: No bootstrap required, opened a new log
I20240502 21:29:41.371563 25966 ts_tablet_manager.cc:1367] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20240502 21:29:41.371642 25969 tablet_bootstrap.cc:654] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:41.372428 25966 raft_consensus.cc:348] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:41.372756 25966 raft_consensus.cc:374] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:41.372875 25966 raft_consensus.cc:729] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Initialized, Role: FOLLOWER
I20240502 21:29:41.373183 25966 consensus_queue.cc:260] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:41.373941 25966 ts_tablet_manager.cc:1398] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:29:41.374306 25966 tablet_bootstrap.cc:492] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced: Bootstrap starting.
I20240502 21:29:41.375716 25969 tablet_bootstrap.cc:492] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c: No bootstrap required, opened a new log
I20240502 21:29:41.375876 25969 ts_tablet_manager.cc:1367] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240502 21:29:41.376032 25966 tablet_bootstrap.cc:654] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:41.376691 25969 raft_consensus.cc:348] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:41.376931 25969 raft_consensus.cc:374] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:41.377041 25969 raft_consensus.cc:729] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Initialized, Role: FOLLOWER
I20240502 21:29:41.377321 25969 consensus_queue.cc:260] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:41.377786 25969 ts_tablet_manager.cc:1398] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:29:41.378310 25966 tablet_bootstrap.cc:492] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced: No bootstrap required, opened a new log
I20240502 21:29:41.378456 25966 ts_tablet_manager.cc:1367] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:29:41.379228 25966 raft_consensus.cc:348] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:41.379434 25966 raft_consensus.cc:374] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:41.379544 25966 raft_consensus.cc:729] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Initialized, Role: FOLLOWER
I20240502 21:29:41.379779 25966 consensus_queue.cc:260] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:41.380378 25966 ts_tablet_manager.cc:1398] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:29:41.572582 25899 raft_consensus.cc:482] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:41.572836 25899 raft_consensus.cc:504] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:41.573451 25899 leader_election.cc:288] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:41.573938 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a378f7a44648c3accc9dcba847040d" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced" is_pre_election: true
I20240502 21:29:41.574111 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a378f7a44648c3accc9dcba847040d" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557" is_pre_election: true
I20240502 21:29:41.574191 25123 raft_consensus.cc:2454] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 0.
I20240502 21:29:41.574354 24991 raft_consensus.cc:2454] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 0.
I20240502 21:29:41.574712 24789 leader_election.cc:302] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:41.574985 25899 raft_consensus.cc:2781] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:41.575141 25899 raft_consensus.cc:482] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:41.575266 25899 raft_consensus.cc:3037] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:41.577309 25899 raft_consensus.cc:504] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:41.577893 25899 leader_election.cc:288] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 1 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:41.578271 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a378f7a44648c3accc9dcba847040d" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:41.578495 25123 raft_consensus.cc:3037] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:41.578421 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a378f7a44648c3accc9dcba847040d" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:41.578648 24991 raft_consensus.cc:3037] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:41.580531 24991 raft_consensus.cc:2454] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 1.
I20240502 21:29:41.580533 25123 raft_consensus.cc:2454] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 1.
I20240502 21:29:41.580952 24789 leader_election.cc:302] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:41.581257 25899 raft_consensus.cc:2781] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:41.581387 25899 raft_consensus.cc:686] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [term 1 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:41.581702 25899 consensus_queue.cc:237] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:41.584182 24712 catalog_manager.cc:5484] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 0 to 1, leader changed from <none> to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 1 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } }
I20240502 21:29:41.592130 25882 raft_consensus.cc:482] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:41.592288 25882 raft_consensus.cc:504] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:41.593057 25882 leader_election.cc:288] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:41.593526 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69afb121cb564de3a9bec5e3f1cc0492" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced" is_pre_election: true
I20240502 21:29:41.593547 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69afb121cb564de3a9bec5e3f1cc0492" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557" is_pre_election: true
I20240502 21:29:41.593796 24992 raft_consensus.cc:2454] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 0.
I20240502 21:29:41.593782 25123 raft_consensus.cc:2454] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 0.
I20240502 21:29:41.594229 25191 leader_election.cc:302] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:41.594493 25882 raft_consensus.cc:2781] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:41.594599 25882 raft_consensus.cc:482] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:41.594707 25882 raft_consensus.cc:3037] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:41.596621 25882 raft_consensus.cc:504] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:41.597128 25882 leader_election.cc:288] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:41.597601 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69afb121cb564de3a9bec5e3f1cc0492" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:41.597637 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69afb121cb564de3a9bec5e3f1cc0492" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:41.597754 25123 raft_consensus.cc:3037] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:41.597848 24992 raft_consensus.cc:3037] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:41.599779 24992 raft_consensus.cc:2454] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 1.
I20240502 21:29:41.599848 25123 raft_consensus.cc:2454] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 1.
I20240502 21:29:41.600212 25191 leader_election.cc:302] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:41.600517 25882 raft_consensus.cc:2781] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:41.600682 25882 raft_consensus.cc:686] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:41.600960 25882 consensus_queue.cc:237] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:41.603448 24712 catalog_manager.cc:5484] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 1 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:41.642642 25909 raft_consensus.cc:482] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:41.642869 25909 raft_consensus.cc:504] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:41.644552 25909 leader_election.cc:288] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:41.653311 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52153a02d80043f0a1840b0e5aadd9b3" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced" is_pre_election: true
I20240502 21:29:41.653620 25123 raft_consensus.cc:2454] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 0.
I20240502 21:29:41.653491 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52153a02d80043f0a1840b0e5aadd9b3" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b7d5b9696b4046b522c880c748fe6c" is_pre_election: true
I20240502 21:29:41.653908 25261 raft_consensus.cc:2454] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 0.
I20240502 21:29:41.654103 24926 leader_election.cc:302] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:41.654526 25909 raft_consensus.cc:2781] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:41.654657 25909 raft_consensus.cc:482] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:41.654748 25909 raft_consensus.cc:3037] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:41.656925 25909 raft_consensus.cc:504] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:41.657449 25909 leader_election.cc:288] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 1 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:41.657858 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52153a02d80043f0a1840b0e5aadd9b3" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:41.657897 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52153a02d80043f0a1840b0e5aadd9b3" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:41.658047 25123 raft_consensus.cc:3037] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:41.658057 25261 raft_consensus.cc:3037] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:41.659852 25123 raft_consensus.cc:2454] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 1.
I20240502 21:29:41.660050 25261 raft_consensus.cc:2454] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 1.
I20240502 21:29:41.660219 24926 leader_election.cc:302] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:41.660528 25909 raft_consensus.cc:2781] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:41.660856 25909 raft_consensus.cc:686] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:41.661475 25909 consensus_queue.cc:237] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:41.664422 24712 catalog_manager.cc:5484] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:42.017854 25882 consensus_queue.cc:1035] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:42.046293 25894 consensus_queue.cc:1035] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:29:42.056463 25988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:42.056797 25988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:42.064177 25988 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:29:42.080147 25899 consensus_queue.cc:1035] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:42.082481 25899 consensus_queue.cc:1035] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240502 21:29:42.104969 25909 consensus_queue.cc:1035] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:42.110769 25909 consensus_queue.cc:1035] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 255.287 ms
time per row: 0.127643 ms
I20240502 21:29:42.492800 24712 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53668:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:29:42.502743 24837 tablet_service.cc:1467] Processing CreateTablet for tablet 8423ff21d93741cdab3093ea454bb532 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c9d9ca99412431db14ae9d0faf500c0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:42.502919 24970 tablet_service.cc:1467] Processing CreateTablet for tablet 8423ff21d93741cdab3093ea454bb532 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c9d9ca99412431db14ae9d0faf500c0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:42.503314 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8423ff21d93741cdab3093ea454bb532. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:42.503440 24970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8423ff21d93741cdab3093ea454bb532. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:42.504262 25102 tablet_service.cc:1467] Processing CreateTablet for tablet 5360f56e93ef486b94bebcddf2edfaaa (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c9d9ca99412431db14ae9d0faf500c0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:42.504384 24835 tablet_service.cc:1467] Processing CreateTablet for tablet 5360f56e93ef486b94bebcddf2edfaaa (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c9d9ca99412431db14ae9d0faf500c0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:42.504424 24971 tablet_service.cc:1467] Processing CreateTablet for tablet 57b40e72e8324b4ba6febf5eda24d333 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c9d9ca99412431db14ae9d0faf500c0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:42.504715 25102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5360f56e93ef486b94bebcddf2edfaaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:42.504724 24835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5360f56e93ef486b94bebcddf2edfaaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:42.504810 24971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57b40e72e8324b4ba6febf5eda24d333. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:42.508019 24836 tablet_service.cc:1467] Processing CreateTablet for tablet 57b40e72e8324b4ba6febf5eda24d333 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c9d9ca99412431db14ae9d0faf500c0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:42.508363 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57b40e72e8324b4ba6febf5eda24d333. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:42.510344 25241 tablet_service.cc:1467] Processing CreateTablet for tablet 8423ff21d93741cdab3093ea454bb532 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c9d9ca99412431db14ae9d0faf500c0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:42.510655 25239 tablet_service.cc:1467] Processing CreateTablet for tablet 57b40e72e8324b4ba6febf5eda24d333 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c9d9ca99412431db14ae9d0faf500c0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:42.510864 25241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8423ff21d93741cdab3093ea454bb532. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:42.511092 25239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57b40e72e8324b4ba6febf5eda24d333. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:42.512084 25240 tablet_service.cc:1467] Processing CreateTablet for tablet 5360f56e93ef486b94bebcddf2edfaaa (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c9d9ca99412431db14ae9d0faf500c0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:42.512671 25240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5360f56e93ef486b94bebcddf2edfaaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:42.522377 26060 tablet_bootstrap.cc:492] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced: Bootstrap starting.
I20240502 21:29:42.523146 26057 tablet_bootstrap.cc:492] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557: Bootstrap starting.
I20240502 21:29:42.525240 26057 tablet_bootstrap.cc:654] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:42.525607 26060 tablet_bootstrap.cc:654] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:42.531992 26060 tablet_bootstrap.cc:492] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced: No bootstrap required, opened a new log
I20240502 21:29:42.532263 26060 ts_tablet_manager.cc:1367] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240502 21:29:42.533213 26060 raft_consensus.cc:348] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:42.533560 26063 tablet_bootstrap.cc:492] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c: Bootstrap starting.
I20240502 21:29:42.533826 26060 raft_consensus.cc:374] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:42.533993 26060 raft_consensus.cc:729] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Initialized, Role: FOLLOWER
I20240502 21:29:42.534356 26060 consensus_queue.cc:260] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:42.535017 26060 ts_tablet_manager.cc:1398] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:29:42.536975 26063 tablet_bootstrap.cc:654] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:42.539397 26063 tablet_bootstrap.cc:492] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c: No bootstrap required, opened a new log
I20240502 21:29:42.539541 26063 ts_tablet_manager.cc:1367] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 21:29:42.540985 26058 tablet_bootstrap.cc:492] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b: Bootstrap starting.
I20240502 21:29:42.540922 26063 raft_consensus.cc:348] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:42.541126 26063 raft_consensus.cc:374] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:42.541231 26063 raft_consensus.cc:729] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Initialized, Role: FOLLOWER
I20240502 21:29:42.541528 26063 consensus_queue.cc:260] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:42.542105 26063 ts_tablet_manager.cc:1398] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:29:42.542376 26063 tablet_bootstrap.cc:492] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c: Bootstrap starting.
I20240502 21:29:42.543128 26058 tablet_bootstrap.cc:654] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:42.544373 26063 tablet_bootstrap.cc:654] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:42.547178 26058 tablet_bootstrap.cc:492] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b: No bootstrap required, opened a new log
I20240502 21:29:42.547344 26058 ts_tablet_manager.cc:1367] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 21:29:42.548362 26058 raft_consensus.cc:348] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:42.548635 26058 raft_consensus.cc:374] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:42.548753 26058 raft_consensus.cc:729] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Initialized, Role: FOLLOWER
I20240502 21:29:42.548804 26063 tablet_bootstrap.cc:492] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c: No bootstrap required, opened a new log
I20240502 21:29:42.549063 26063 ts_tablet_manager.cc:1367] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240502 21:29:42.549077 26058 consensus_queue.cc:260] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:42.549764 26058 ts_tablet_manager.cc:1398] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 21:29:42.549870 26063 raft_consensus.cc:348] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:42.550047 26058 tablet_bootstrap.cc:492] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b: Bootstrap starting.
I20240502 21:29:42.550104 26063 raft_consensus.cc:374] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:42.550191 26063 raft_consensus.cc:729] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Initialized, Role: FOLLOWER
I20240502 21:29:42.550451 26063 consensus_queue.cc:260] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:42.551045 26063 ts_tablet_manager.cc:1398] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:29:42.551352 26063 tablet_bootstrap.cc:492] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c: Bootstrap starting.
I20240502 21:29:42.551551 26057 tablet_bootstrap.cc:492] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557: No bootstrap required, opened a new log
I20240502 21:29:42.552181 26057 ts_tablet_manager.cc:1367] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.011s
I20240502 21:29:42.552920 26063 tablet_bootstrap.cc:654] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:42.553373 26057 raft_consensus.cc:348] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:42.553599 26058 tablet_bootstrap.cc:654] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:42.553730 26057 raft_consensus.cc:374] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:42.553929 26057 raft_consensus.cc:729] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Initialized, Role: FOLLOWER
I20240502 21:29:42.554360 26057 consensus_queue.cc:260] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:42.555141 26057 ts_tablet_manager.cc:1398] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:29:42.555526 26057 tablet_bootstrap.cc:492] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557: Bootstrap starting.
I20240502 21:29:42.555855 26063 tablet_bootstrap.cc:492] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c: No bootstrap required, opened a new log
I20240502 21:29:42.556000 26063 ts_tablet_manager.cc:1367] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 21:29:42.556768 26063 raft_consensus.cc:348] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:42.556982 26063 raft_consensus.cc:374] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:42.557104 26063 raft_consensus.cc:729] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Initialized, Role: FOLLOWER
I20240502 21:29:42.557528 26057 tablet_bootstrap.cc:654] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:42.557379 26063 consensus_queue.cc:260] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:42.558501 26063 ts_tablet_manager.cc:1398] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:29:42.565716 26058 tablet_bootstrap.cc:492] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b: No bootstrap required, opened a new log
I20240502 21:29:42.565743 26057 tablet_bootstrap.cc:492] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557: No bootstrap required, opened a new log
I20240502 21:29:42.565867 26058 ts_tablet_manager.cc:1367] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.008s
I20240502 21:29:42.565884 26057 ts_tablet_manager.cc:1367] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20240502 21:29:42.566409 26057 raft_consensus.cc:348] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:42.566581 26057 raft_consensus.cc:374] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:42.566551 26058 raft_consensus.cc:348] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:42.566761 26058 raft_consensus.cc:374] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:42.566823 26058 raft_consensus.cc:729] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Initialized, Role: FOLLOWER
I20240502 21:29:42.567051 26058 consensus_queue.cc:260] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:42.566659 26057 raft_consensus.cc:729] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Initialized, Role: FOLLOWER
I20240502 21:29:42.567433 26057 consensus_queue.cc:260] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:42.567613 26058 ts_tablet_manager.cc:1398] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:29:42.567984 26057 ts_tablet_manager.cc:1398] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:29:42.568557 26058 tablet_bootstrap.cc:492] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b: Bootstrap starting.
I20240502 21:29:42.570696 26058 tablet_bootstrap.cc:654] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:42.576797 26058 tablet_bootstrap.cc:492] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b: No bootstrap required, opened a new log
I20240502 21:29:42.576959 26058 ts_tablet_manager.cc:1367] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20240502 21:29:42.577728 26058 raft_consensus.cc:348] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:42.577958 26058 raft_consensus.cc:374] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:42.578047 26058 raft_consensus.cc:729] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Initialized, Role: FOLLOWER
I20240502 21:29:42.578275 26058 consensus_queue.cc:260] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:42.579007 26058 ts_tablet_manager.cc:1398] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 21:29:42.623826 26007 raft_consensus.cc:482] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:42.624038 26007 raft_consensus.cc:504] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:42.624855 26007 leader_election.cc:288] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:42.625444 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8423ff21d93741cdab3093ea454bb532" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557" is_pre_election: true
I20240502 21:29:42.625761 24991 raft_consensus.cc:2454] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 0.
I20240502 21:29:42.626282 24789 leader_election.cc:302] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:42.626595 26007 raft_consensus.cc:2781] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:42.626765 26007 raft_consensus.cc:482] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:42.626890 26007 raft_consensus.cc:3037] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:42.629570 26007 raft_consensus.cc:504] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:42.630333 26007 leader_election.cc:288] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 1 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:42.630733 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8423ff21d93741cdab3093ea454bb532" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:42.630949 24991 raft_consensus.cc:3037] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:42.632408 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8423ff21d93741cdab3093ea454bb532" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b7d5b9696b4046b522c880c748fe6c" is_pre_election: true
I20240502 21:29:42.632524 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8423ff21d93741cdab3093ea454bb532" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:42.632712 25261 raft_consensus.cc:2454] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 0.
I20240502 21:29:42.633435 24991 raft_consensus.cc:2454] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 1.
I20240502 21:29:42.633733 25986 raft_consensus.cc:482] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:42.633843 24789 leader_election.cc:302] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b; no voters: 45b7d5b9696b4046b522c880c748fe6c
I20240502 21:29:42.633889 25986 raft_consensus.cc:504] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:42.634153 26007 raft_consensus.cc:2781] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:42.634320 26007 raft_consensus.cc:686] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 1 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:42.634821 25986 leader_election.cc:288] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:42.635061 26007 consensus_queue.cc:237] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:42.635340 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b40e72e8324b4ba6febf5eda24d333" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b7d5b9696b4046b522c880c748fe6c" is_pre_election: true
I20240502 21:29:42.635610 25260 raft_consensus.cc:2454] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 0.
I20240502 21:29:42.636221 24927 leader_election.cc:302] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:42.636665 25986 raft_consensus.cc:2781] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:42.636804 25986 raft_consensus.cc:482] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:42.636910 25986 raft_consensus.cc:3037] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:42.639423 25986 raft_consensus.cc:504] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:42.639537 24708 catalog_manager.cc:5484] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 0 to 1, leader changed from <none> to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 1 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:42.640102 25986 leader_election.cc:288] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 1 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:42.640698 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b40e72e8324b4ba6febf5eda24d333" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:42.640925 25260 raft_consensus.cc:3037] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:42.643689 25260 raft_consensus.cc:2454] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 1.
I20240502 21:29:42.644155 24927 leader_election.cc:302] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:42.644490 25986 raft_consensus.cc:2781] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:42.644686 25986 raft_consensus.cc:686] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:42.645004 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b40e72e8324b4ba6febf5eda24d333" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:42.645051 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b40e72e8324b4ba6febf5eda24d333" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b" is_pre_election: true
I20240502 21:29:42.645244 24857 raft_consensus.cc:3037] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:42.645200 25986 consensus_queue.cc:237] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:42.647233 24857 raft_consensus.cc:2454] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 1.
I20240502 21:29:42.647954 24708 catalog_manager.cc:5484] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:42.720494 25996 raft_consensus.cc:482] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:42.720712 25996 raft_consensus.cc:504] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:42.722287 25894 raft_consensus.cc:482] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:42.722491 25894 raft_consensus.cc:504] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:42.722680 25996 leader_election.cc:288] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:42.723385 25894 leader_election.cc:288] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:42.724839 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5360f56e93ef486b94bebcddf2edfaaa" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b" is_pre_election: true
I20240502 21:29:42.725150 24857 raft_consensus.cc:2454] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 0.
I20240502 21:29:42.725778 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5360f56e93ef486b94bebcddf2edfaaa" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced" is_pre_election: true
I20240502 21:29:42.726018 25123 raft_consensus.cc:2454] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 0.
I20240502 21:29:42.726415 25191 leader_election.cc:302] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:42.726785 25894 raft_consensus.cc:2781] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:42.726891 25894 raft_consensus.cc:482] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:42.726984 25894 raft_consensus.cc:3037] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:42.729116 25894 raft_consensus.cc:504] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:42.729776 25894 leader_election.cc:288] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:42.730504 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5360f56e93ef486b94bebcddf2edfaaa" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:42.730552 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5360f56e93ef486b94bebcddf2edfaaa" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:42.730756 25123 raft_consensus.cc:3037] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:42.730765 24857 raft_consensus.cc:3037] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:42.731467 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5360f56e93ef486b94bebcddf2edfaaa" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b" is_pre_election: true
I20240502 21:29:42.733091 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5360f56e93ef486b94bebcddf2edfaaa" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b7d5b9696b4046b522c880c748fe6c" is_pre_election: true
I20240502 21:29:42.733368 25260 raft_consensus.cc:2379] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5bf90e514781418da921c046aa5d9ced in current term 1: Already voted for candidate 45b7d5b9696b4046b522c880c748fe6c in this term.
I20240502 21:29:42.733377 25123 raft_consensus.cc:2454] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 1.
I20240502 21:29:42.733686 24857 raft_consensus.cc:2454] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 1.
I20240502 21:29:42.733820 25191 leader_election.cc:302] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced; no voters:
I20240502 21:29:42.733907 25059 leader_election.cc:302] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced; no voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b
I20240502 21:29:42.734148 25894 raft_consensus.cc:2781] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:42.734319 25894 raft_consensus.cc:686] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:42.734524 25996 raft_consensus.cc:2726] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:29:42.734614 25894 consensus_queue.cc:237] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:42.737512 24708 catalog_manager.cc:5484] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 1 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:43.050859 25260 raft_consensus.cc:3037] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:43.053781 26007 consensus_queue.cc:1035] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:43.061836 26007 consensus_queue.cc:1035] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:29:43.117151 26079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:43.117543 26079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:43.125748 26079 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:29:43.148672 25894 consensus_queue.cc:1035] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:43.154551 26037 consensus_queue.cc:1035] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:43.164068 25986 consensus_queue.cc:1035] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240502 21:29:43.170032 26103 consensus_queue.cc:1035] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 249.55 ms
time per row: 0.124775 ms
I20240502 21:29:43.542455 24708 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53668:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:29:43.552305 24836 tablet_service.cc:1467] Processing CreateTablet for tablet c574664a57444c95b70d754cc46e313e (DEFAULT_TABLE table=rebalance_test_table_3 [id=ff7b441e53054a4fa3e121f0bae5792a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:43.552606 25102 tablet_service.cc:1467] Processing CreateTablet for tablet c574664a57444c95b70d754cc46e313e (DEFAULT_TABLE table=rebalance_test_table_3 [id=ff7b441e53054a4fa3e121f0bae5792a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:43.552829 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c574664a57444c95b70d754cc46e313e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:43.553087 25102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c574664a57444c95b70d754cc46e313e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:43.556011 24835 tablet_service.cc:1467] Processing CreateTablet for tablet e145bbec053d4028ad7bd38d25dd4315 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ff7b441e53054a4fa3e121f0bae5792a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:43.556092 24970 tablet_service.cc:1467] Processing CreateTablet for tablet c574664a57444c95b70d754cc46e313e (DEFAULT_TABLE table=rebalance_test_table_3 [id=ff7b441e53054a4fa3e121f0bae5792a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:43.556362 24835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e145bbec053d4028ad7bd38d25dd4315. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:43.556432 24970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c574664a57444c95b70d754cc46e313e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:43.560678 25239 tablet_service.cc:1467] Processing CreateTablet for tablet e145bbec053d4028ad7bd38d25dd4315 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ff7b441e53054a4fa3e121f0bae5792a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:43.561101 25239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e145bbec053d4028ad7bd38d25dd4315. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:43.561211 24971 tablet_service.cc:1467] Processing CreateTablet for tablet 6f25dfc6b0c04d878cd3db58d37cddec (DEFAULT_TABLE table=rebalance_test_table_3 [id=ff7b441e53054a4fa3e121f0bae5792a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:43.561627 24971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f25dfc6b0c04d878cd3db58d37cddec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:43.563936 25104 tablet_service.cc:1467] Processing CreateTablet for tablet e145bbec053d4028ad7bd38d25dd4315 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ff7b441e53054a4fa3e121f0bae5792a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:43.564311 25104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e145bbec053d4028ad7bd38d25dd4315. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:43.568042 25103 tablet_service.cc:1467] Processing CreateTablet for tablet 6f25dfc6b0c04d878cd3db58d37cddec (DEFAULT_TABLE table=rebalance_test_table_3 [id=ff7b441e53054a4fa3e121f0bae5792a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:43.568519 25103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f25dfc6b0c04d878cd3db58d37cddec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:43.572105 25240 tablet_service.cc:1467] Processing CreateTablet for tablet 6f25dfc6b0c04d878cd3db58d37cddec (DEFAULT_TABLE table=rebalance_test_table_3 [id=ff7b441e53054a4fa3e121f0bae5792a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:43.572587 25240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f25dfc6b0c04d878cd3db58d37cddec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:43.576444 26147 tablet_bootstrap.cc:492] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557: Bootstrap starting.
I20240502 21:29:43.577425 26148 tablet_bootstrap.cc:492] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced: Bootstrap starting.
I20240502 21:29:43.577857 26146 tablet_bootstrap.cc:492] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b: Bootstrap starting.
I20240502 21:29:43.578534 26147 tablet_bootstrap.cc:654] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:43.579259 26148 tablet_bootstrap.cc:654] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:43.581420 26146 tablet_bootstrap.cc:654] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:43.588907 26147 tablet_bootstrap.cc:492] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557: No bootstrap required, opened a new log
I20240502 21:29:43.589583 26147 ts_tablet_manager.cc:1367] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.004s
I20240502 21:29:43.590705 26147 raft_consensus.cc:348] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:43.592069 26147 raft_consensus.cc:374] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:43.592360 26147 raft_consensus.cc:729] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Initialized, Role: FOLLOWER
I20240502 21:29:43.592964 26147 consensus_queue.cc:260] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:43.596580 26147 ts_tablet_manager.cc:1398] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 21:29:43.599507 26147 tablet_bootstrap.cc:492] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557: Bootstrap starting.
I20240502 21:29:43.602931 26146 tablet_bootstrap.cc:492] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b: No bootstrap required, opened a new log
I20240502 21:29:43.603278 26146 ts_tablet_manager.cc:1367] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.012s
I20240502 21:29:43.603310 26147 tablet_bootstrap.cc:654] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:43.604758 26146 raft_consensus.cc:348] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:43.605208 26146 raft_consensus.cc:374] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:43.605396 26146 raft_consensus.cc:729] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Initialized, Role: FOLLOWER
I20240502 21:29:43.605825 26146 consensus_queue.cc:260] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:43.606956 26146 ts_tablet_manager.cc:1398] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240502 21:29:43.607765 26149 tablet_bootstrap.cc:492] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c: Bootstrap starting.
I20240502 21:29:43.609215 26146 tablet_bootstrap.cc:492] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b: Bootstrap starting.
I20240502 21:29:43.610476 26149 tablet_bootstrap.cc:654] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:43.611773 26146 tablet_bootstrap.cc:654] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:43.612080 26147 tablet_bootstrap.cc:492] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557: No bootstrap required, opened a new log
I20240502 21:29:43.612331 26147 ts_tablet_manager.cc:1367] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.001s
I20240502 21:29:43.613157 26147 raft_consensus.cc:348] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:43.613471 26147 raft_consensus.cc:374] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:43.613590 26147 raft_consensus.cc:729] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Initialized, Role: FOLLOWER
I20240502 21:29:43.613894 26147 consensus_queue.cc:260] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:43.616747 26147 ts_tablet_manager.cc:1398] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:29:43.617367 26148 tablet_bootstrap.cc:492] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced: No bootstrap required, opened a new log
I20240502 21:29:43.617868 26148 ts_tablet_manager.cc:1367] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced: Time spent bootstrapping tablet: real 0.041s user 0.026s sys 0.013s
I20240502 21:29:43.618872 26148 raft_consensus.cc:348] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:43.619216 26148 raft_consensus.cc:374] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:43.619385 26148 raft_consensus.cc:729] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Initialized, Role: FOLLOWER
I20240502 21:29:43.619760 26148 consensus_queue.cc:260] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:43.620702 26148 ts_tablet_manager.cc:1398] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:29:43.620743 26149 tablet_bootstrap.cc:492] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c: No bootstrap required, opened a new log
I20240502 21:29:43.620884 26149 ts_tablet_manager.cc:1367] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.003s
I20240502 21:29:43.621045 26148 tablet_bootstrap.cc:492] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced: Bootstrap starting.
I20240502 21:29:43.621789 26149 raft_consensus.cc:348] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:43.622130 26149 raft_consensus.cc:374] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:43.622272 26149 raft_consensus.cc:729] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Initialized, Role: FOLLOWER
I20240502 21:29:43.622849 26148 tablet_bootstrap.cc:654] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:43.622610 26149 consensus_queue.cc:260] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:43.623723 26149 ts_tablet_manager.cc:1398] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:29:43.624980 26146 tablet_bootstrap.cc:492] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b: No bootstrap required, opened a new log
I20240502 21:29:43.625355 26146 ts_tablet_manager.cc:1367] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.006s
I20240502 21:29:43.625649 26148 tablet_bootstrap.cc:492] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced: No bootstrap required, opened a new log
I20240502 21:29:43.625803 26148 ts_tablet_manager.cc:1367] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240502 21:29:43.626242 26146 raft_consensus.cc:348] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:43.626477 26146 raft_consensus.cc:374] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:43.626575 26146 raft_consensus.cc:729] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Initialized, Role: FOLLOWER
I20240502 21:29:43.626498 26148 raft_consensus.cc:348] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:43.626698 26148 raft_consensus.cc:374] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:43.626775 26148 raft_consensus.cc:729] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Initialized, Role: FOLLOWER
I20240502 21:29:43.626832 26146 consensus_queue.cc:260] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:43.627007 26148 consensus_queue.cc:260] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:43.627812 26148 ts_tablet_manager.cc:1398] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 21:29:43.628152 26146 ts_tablet_manager.cc:1398] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:29:43.628679 26149 tablet_bootstrap.cc:492] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c: Bootstrap starting.
I20240502 21:29:43.630499 26149 tablet_bootstrap.cc:654] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:43.634608 26148 tablet_bootstrap.cc:492] T e145bbec053d4028ad7bd38d25dd4315 P 5bf90e514781418da921c046aa5d9ced: Bootstrap starting.
I20240502 21:29:43.636958 26148 tablet_bootstrap.cc:654] T e145bbec053d4028ad7bd38d25dd4315 P 5bf90e514781418da921c046aa5d9ced: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:43.639004 26148 tablet_bootstrap.cc:492] T e145bbec053d4028ad7bd38d25dd4315 P 5bf90e514781418da921c046aa5d9ced: No bootstrap required, opened a new log
I20240502 21:29:43.639192 26148 ts_tablet_manager.cc:1367] T e145bbec053d4028ad7bd38d25dd4315 P 5bf90e514781418da921c046aa5d9ced: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 21:29:43.639245 26149 tablet_bootstrap.cc:492] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c: No bootstrap required, opened a new log
I20240502 21:29:43.639461 26149 ts_tablet_manager.cc:1367] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.002s
I20240502 21:29:43.639807 26148 raft_consensus.cc:348] T e145bbec053d4028ad7bd38d25dd4315 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:43.639957 26148 raft_consensus.cc:374] T e145bbec053d4028ad7bd38d25dd4315 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:43.640089 26148 raft_consensus.cc:729] T e145bbec053d4028ad7bd38d25dd4315 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Initialized, Role: FOLLOWER
I20240502 21:29:43.640203 26149 raft_consensus.cc:348] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:43.640411 26149 raft_consensus.cc:374] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:43.640486 26149 raft_consensus.cc:729] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Initialized, Role: FOLLOWER
I20240502 21:29:43.640364 26148 consensus_queue.cc:260] T e145bbec053d4028ad7bd38d25dd4315 P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:43.640875 26148 ts_tablet_manager.cc:1398] T e145bbec053d4028ad7bd38d25dd4315 P 5bf90e514781418da921c046aa5d9ced: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:29:43.640690 26149 consensus_queue.cc:260] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:43.642100 26149 ts_tablet_manager.cc:1398] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:29:43.659595 25996 raft_consensus.cc:482] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:43.659770 25996 raft_consensus.cc:504] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:43.660421 25996 leader_election.cc:288] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:43.660991 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f25dfc6b0c04d878cd3db58d37cddec" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b7d5b9696b4046b522c880c748fe6c" is_pre_election: true
I20240502 21:29:43.661244 25260 raft_consensus.cc:2454] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 0.
I20240502 21:29:43.661620 25059 leader_election.cc:302] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced; no voters:
I20240502 21:29:43.661923 25996 raft_consensus.cc:2781] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:43.662045 25996 raft_consensus.cc:482] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:43.662145 25996 raft_consensus.cc:3037] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:43.664867 25996 raft_consensus.cc:504] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:43.665481 25996 leader_election.cc:288] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:43.666329 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f25dfc6b0c04d878cd3db58d37cddec" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:43.666525 25260 raft_consensus.cc:3037] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:43.667398 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f25dfc6b0c04d878cd3db58d37cddec" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557" is_pre_election: true
I20240502 21:29:43.667456 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f25dfc6b0c04d878cd3db58d37cddec" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:43.667687 24991 raft_consensus.cc:2454] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 0.
I20240502 21:29:43.668803 25260 raft_consensus.cc:2454] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 1.
I20240502 21:29:43.669164 25059 leader_election.cc:302] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced; no voters: 64d97dd1674d43ccadd7b4a38eeff557
I20240502 21:29:43.669458 25996 raft_consensus.cc:2781] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:43.669847 25996 raft_consensus.cc:686] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [term 1 LEADER]: Becoming Leader. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:43.670286 25996 consensus_queue.cc:237] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:43.673453 24708 catalog_manager.cc:5484] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bf90e514781418da921c046aa5d9ced (127.29.82.67). New cstate: current_term: 1 leader_uuid: "5bf90e514781418da921c046aa5d9ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:43.724424 26103 raft_consensus.cc:482] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:43.724731 26103 raft_consensus.cc:504] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:43.725693 26103 leader_election.cc:288] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:43.726197 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c574664a57444c95b70d754cc46e313e" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b" is_pre_election: true
I20240502 21:29:43.726296 25122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c574664a57444c95b70d754cc46e313e" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced" is_pre_election: true
I20240502 21:29:43.726480 24857 raft_consensus.cc:2454] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 0.
I20240502 21:29:43.726585 25122 raft_consensus.cc:2454] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 0.
I20240502 21:29:43.726971 24926 leader_election.cc:302] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:43.727277 26103 raft_consensus.cc:2781] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:43.727383 26103 raft_consensus.cc:482] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:43.727488 26103 raft_consensus.cc:3037] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:43.729976 26103 raft_consensus.cc:504] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:43.730489 26103 leader_election.cc:288] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 1 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:43.730995 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c574664a57444c95b70d754cc46e313e" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:43.731096 25122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c574664a57444c95b70d754cc46e313e" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:43.731194 24857 raft_consensus.cc:3037] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:43.731284 25122 raft_consensus.cc:3037] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:43.733237 24857 raft_consensus.cc:2454] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 1.
I20240502 21:29:43.733271 25122 raft_consensus.cc:2454] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 1.
I20240502 21:29:43.733624 24926 leader_election.cc:302] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:43.733882 26103 raft_consensus.cc:2781] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:43.734055 26103 raft_consensus.cc:686] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:43.734320 26103 consensus_queue.cc:237] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:43.737228 24708 catalog_manager.cc:5484] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:43.827715 26037 raft_consensus.cc:482] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:43.827929 26037 raft_consensus.cc:504] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:43.828503 26037 leader_election.cc:288] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:43.829052 25122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e145bbec053d4028ad7bd38d25dd4315" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced" is_pre_election: true
I20240502 21:29:43.829073 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e145bbec053d4028ad7bd38d25dd4315" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b" is_pre_election: true
I20240502 21:29:43.829339 25122 raft_consensus.cc:2454] T e145bbec053d4028ad7bd38d25dd4315 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 0.
I20240502 21:29:43.829339 24857 raft_consensus.cc:2454] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 0.
I20240502 21:29:43.829771 25191 leader_election.cc:302] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:43.830042 26037 raft_consensus.cc:2781] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:43.830148 26037 raft_consensus.cc:482] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:43.830256 26037 raft_consensus.cc:3037] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:43.832141 26037 raft_consensus.cc:504] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:43.832691 26037 leader_election.cc:288] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:43.833046 25122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e145bbec053d4028ad7bd38d25dd4315" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:43.833297 25122 raft_consensus.cc:3037] T e145bbec053d4028ad7bd38d25dd4315 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:43.833241 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e145bbec053d4028ad7bd38d25dd4315" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:43.833655 24857 raft_consensus.cc:3037] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:43.835389 25122 raft_consensus.cc:2454] T e145bbec053d4028ad7bd38d25dd4315 P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 1.
I20240502 21:29:43.835789 25191 leader_election.cc:302] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced; no voters:
I20240502 21:29:43.835832 24857 raft_consensus.cc:2454] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 1.
I20240502 21:29:43.836058 26037 raft_consensus.cc:2781] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:43.836282 26037 raft_consensus.cc:686] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:43.836557 26037 consensus_queue.cc:237] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:43.839049 24708 catalog_manager.cc:5484] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 1 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:44.117393 24991 raft_consensus.cc:3037] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:44.120265 25996 consensus_queue.cc:1035] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:44.125396 25996 consensus_queue.cc:1035] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:29:44.210852 26164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:44.211170 26164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:44.219110 26164 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 21:29:44.283754 26037 consensus_queue.cc:1035] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:44.289286 25894 consensus_queue.cc:1035] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:44.292583 26103 consensus_queue.cc:1035] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:44.302290 26103 consensus_queue.cc:1035] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 345.151 ms
time per row: 0.172576 ms
I20240502 21:29:44.707221 24708 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53668:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:29:44.715873 25103 tablet_service.cc:1467] Processing CreateTablet for tablet 175511733b714d2ea3cf877a0e5df418 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e5478f9916ad4294bfafd3e6735ea929]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:44.716070 24835 tablet_service.cc:1467] Processing CreateTablet for tablet 175511733b714d2ea3cf877a0e5df418 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e5478f9916ad4294bfafd3e6735ea929]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:44.716446 25103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 175511733b714d2ea3cf877a0e5df418. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:44.716557 24835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 175511733b714d2ea3cf877a0e5df418. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:44.716928 25239 tablet_service.cc:1467] Processing CreateTablet for tablet fed2d260d00d470e95e9a62cc71ba94f (DEFAULT_TABLE table=rebalance_test_table_4 [id=e5478f9916ad4294bfafd3e6735ea929]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:44.717353 25239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fed2d260d00d470e95e9a62cc71ba94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:44.718099 24971 tablet_service.cc:1467] Processing CreateTablet for tablet fed2d260d00d470e95e9a62cc71ba94f (DEFAULT_TABLE table=rebalance_test_table_4 [id=e5478f9916ad4294bfafd3e6735ea929]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:44.718511 24971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fed2d260d00d470e95e9a62cc71ba94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:44.723588 25104 tablet_service.cc:1467] Processing CreateTablet for tablet acf184a70440492c80cfaea7d80323de (DEFAULT_TABLE table=rebalance_test_table_4 [id=e5478f9916ad4294bfafd3e6735ea929]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:44.724050 25104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acf184a70440492c80cfaea7d80323de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:44.724045 24970 tablet_service.cc:1467] Processing CreateTablet for tablet 175511733b714d2ea3cf877a0e5df418 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e5478f9916ad4294bfafd3e6735ea929]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:44.724587 24970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 175511733b714d2ea3cf877a0e5df418. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:44.728047 25240 tablet_service.cc:1467] Processing CreateTablet for tablet acf184a70440492c80cfaea7d80323de (DEFAULT_TABLE table=rebalance_test_table_4 [id=e5478f9916ad4294bfafd3e6735ea929]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:44.728454 25240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acf184a70440492c80cfaea7d80323de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:44.729656 26235 tablet_bootstrap.cc:492] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced: Bootstrap starting.
I20240502 21:29:44.731109 24836 tablet_service.cc:1467] Processing CreateTablet for tablet fed2d260d00d470e95e9a62cc71ba94f (DEFAULT_TABLE table=rebalance_test_table_4 [id=e5478f9916ad4294bfafd3e6735ea929]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:44.731232 26236 tablet_bootstrap.cc:492] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b: Bootstrap starting.
I20240502 21:29:44.731475 26235 tablet_bootstrap.cc:654] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:44.731535 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fed2d260d00d470e95e9a62cc71ba94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:44.733028 26236 tablet_bootstrap.cc:654] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:44.736085 24837 tablet_service.cc:1467] Processing CreateTablet for tablet acf184a70440492c80cfaea7d80323de (DEFAULT_TABLE table=rebalance_test_table_4 [id=e5478f9916ad4294bfafd3e6735ea929]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:44.736544 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acf184a70440492c80cfaea7d80323de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:44.740146 26237 tablet_bootstrap.cc:492] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c: Bootstrap starting.
I20240502 21:29:44.742324 26237 tablet_bootstrap.cc:654] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:44.751147 26237 tablet_bootstrap.cc:492] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c: No bootstrap required, opened a new log
I20240502 21:29:44.751411 26237 ts_tablet_manager.cc:1367] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240502 21:29:44.752522 26237 raft_consensus.cc:348] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:44.752940 26237 raft_consensus.cc:374] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:44.753137 26237 raft_consensus.cc:729] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Initialized, Role: FOLLOWER
I20240502 21:29:44.753545 26237 consensus_queue.cc:260] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:44.754576 26237 ts_tablet_manager.cc:1398] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:29:44.755008 26237 tablet_bootstrap.cc:492] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c: Bootstrap starting.
I20240502 21:29:44.756759 26237 tablet_bootstrap.cc:654] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:44.758141 26242 tablet_bootstrap.cc:492] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557: Bootstrap starting.
I20240502 21:29:44.761044 26242 tablet_bootstrap.cc:654] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:44.763383 26235 tablet_bootstrap.cc:492] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced: No bootstrap required, opened a new log
I20240502 21:29:44.763556 26235 ts_tablet_manager.cc:1367] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced: Time spent bootstrapping tablet: real 0.034s user 0.018s sys 0.000s
I20240502 21:29:44.764122 26236 tablet_bootstrap.cc:492] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b: No bootstrap required, opened a new log
I20240502 21:29:44.764317 26236 ts_tablet_manager.cc:1367] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.023s
I20240502 21:29:44.764703 26237 tablet_bootstrap.cc:492] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c: No bootstrap required, opened a new log
I20240502 21:29:44.764921 26237 ts_tablet_manager.cc:1367] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20240502 21:29:44.765018 26242 tablet_bootstrap.cc:492] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557: No bootstrap required, opened a new log
I20240502 21:29:44.765223 26242 ts_tablet_manager.cc:1367] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20240502 21:29:44.765164 26236 raft_consensus.cc:348] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:44.765412 26236 raft_consensus.cc:374] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:44.765560 26236 raft_consensus.cc:729] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Initialized, Role: FOLLOWER
I20240502 21:29:44.765690 26237 raft_consensus.cc:348] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:44.765928 26237 raft_consensus.cc:374] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:44.765622 26235 raft_consensus.cc:348] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:44.765892 26236 consensus_queue.cc:260] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:44.766048 26237 raft_consensus.cc:729] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Initialized, Role: FOLLOWER
I20240502 21:29:44.766213 26235 raft_consensus.cc:374] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:44.766361 26235 raft_consensus.cc:729] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Initialized, Role: FOLLOWER
I20240502 21:29:44.766359 26237 consensus_queue.cc:260] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:44.766618 26236 ts_tablet_manager.cc:1398] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:29:44.766688 26235 consensus_queue.cc:260] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:44.767098 26237 ts_tablet_manager.cc:1398] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:29:44.767382 26235 ts_tablet_manager.cc:1398] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 21:29:44.768443 26236 tablet_bootstrap.cc:492] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b: Bootstrap starting.
I20240502 21:29:44.766126 26242 raft_consensus.cc:348] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:44.768654 26242 raft_consensus.cc:374] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:44.768797 26242 raft_consensus.cc:729] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Initialized, Role: FOLLOWER
I20240502 21:29:44.769109 26242 consensus_queue.cc:260] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:44.769735 26242 ts_tablet_manager.cc:1398] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 21:29:44.770697 26242 tablet_bootstrap.cc:492] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557: Bootstrap starting.
I20240502 21:29:44.771092 26236 tablet_bootstrap.cc:654] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:44.772338 26242 tablet_bootstrap.cc:654] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:44.772357 26235 tablet_bootstrap.cc:492] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced: Bootstrap starting.
I20240502 21:29:44.773258 26224 raft_consensus.cc:482] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:44.773399 26224 raft_consensus.cc:504] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:44.774263 26224 leader_election.cc:288] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:44.774791 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf184a70440492c80cfaea7d80323de" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced" is_pre_election: true
I20240502 21:29:44.774963 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf184a70440492c80cfaea7d80323de" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b" is_pre_election: true
W20240502 21:29:44.775977 25191 leader_election.cc:341] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:29:44.776333 25191 leader_election.cc:341] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:29:44.776497 25191 leader_election.cc:302] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c; no voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b
I20240502 21:29:44.777216 26224 raft_consensus.cc:2726] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:29:44.777683 26235 tablet_bootstrap.cc:654] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:44.778939 26242 tablet_bootstrap.cc:492] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557: No bootstrap required, opened a new log
I20240502 21:29:44.778990 26236 tablet_bootstrap.cc:492] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b: No bootstrap required, opened a new log
I20240502 21:29:44.779105 26242 ts_tablet_manager.cc:1367] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 21:29:44.779210 26236 ts_tablet_manager.cc:1367] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.002s
I20240502 21:29:44.779672 26235 tablet_bootstrap.cc:492] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced: No bootstrap required, opened a new log
I20240502 21:29:44.779757 26242 raft_consensus.cc:348] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:44.779844 26235 ts_tablet_manager.cc:1367] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20240502 21:29:44.779939 26242 raft_consensus.cc:374] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:44.780025 26242 raft_consensus.cc:729] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Initialized, Role: FOLLOWER
I20240502 21:29:44.780323 26242 consensus_queue.cc:260] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:44.780779 26235 raft_consensus.cc:348] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:44.781142 26235 raft_consensus.cc:374] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:44.781270 26235 raft_consensus.cc:729] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Initialized, Role: FOLLOWER
I20240502 21:29:44.781299 26242 ts_tablet_manager.cc:1398] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:29:44.780511 26236 raft_consensus.cc:348] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:44.781879 26236 raft_consensus.cc:374] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:44.781615 26235 consensus_queue.cc:260] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:44.782032 26236 raft_consensus.cc:729] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Initialized, Role: FOLLOWER
I20240502 21:29:44.782459 26235 ts_tablet_manager.cc:1398] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:29:44.782339 26236 consensus_queue.cc:260] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:44.783638 26236 ts_tablet_manager.cc:1398] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:29:44.783972 26236 tablet_bootstrap.cc:492] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b: Bootstrap starting.
I20240502 21:29:44.785722 26236 tablet_bootstrap.cc:654] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:44.788142 26236 tablet_bootstrap.cc:492] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b: No bootstrap required, opened a new log
I20240502 21:29:44.788249 26236 ts_tablet_manager.cc:1367] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:29:44.789199 26236 raft_consensus.cc:348] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:44.789342 26236 raft_consensus.cc:374] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:44.789441 26236 raft_consensus.cc:729] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Initialized, Role: FOLLOWER
I20240502 21:29:44.789717 26236 consensus_queue.cc:260] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:44.790184 26236 ts_tablet_manager.cc:1398] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:29:44.829262 26224 raft_consensus.cc:482] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:44.829476 26224 raft_consensus.cc:504] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:44.830094 26224 leader_election.cc:288] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:44.830574 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fed2d260d00d470e95e9a62cc71ba94f" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b" is_pre_election: true
I20240502 21:29:44.830693 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fed2d260d00d470e95e9a62cc71ba94f" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557" is_pre_election: true
I20240502 21:29:44.830871 24857 raft_consensus.cc:2454] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 0.
I20240502 21:29:44.830960 24991 raft_consensus.cc:2454] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 0.
I20240502 21:29:44.831312 25191 leader_election.cc:302] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:44.831597 26224 raft_consensus.cc:2781] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:44.831733 26224 raft_consensus.cc:482] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:44.831830 26224 raft_consensus.cc:3037] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:44.833730 26224 raft_consensus.cc:504] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:44.834261 26224 leader_election.cc:288] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:44.834635 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fed2d260d00d470e95e9a62cc71ba94f" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:44.834744 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fed2d260d00d470e95e9a62cc71ba94f" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:44.834867 24857 raft_consensus.cc:3037] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:44.834983 24991 raft_consensus.cc:3037] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:44.837774 24991 raft_consensus.cc:2454] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 1.
I20240502 21:29:44.837788 24857 raft_consensus.cc:2454] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 1.
I20240502 21:29:44.838176 25191 leader_election.cc:302] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:44.838469 26224 raft_consensus.cc:2781] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:44.838624 26224 raft_consensus.cc:686] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:44.838913 26224 consensus_queue.cc:237] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:44.841795 24705 catalog_manager.cc:5484] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 1 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:44.908567 26007 raft_consensus.cc:482] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:44.908829 26007 raft_consensus.cc:504] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:44.909516 26007 leader_election.cc:288] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:44.910018 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf184a70440492c80cfaea7d80323de" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b7d5b9696b4046b522c880c748fe6c" is_pre_election: true
I20240502 21:29:44.910050 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf184a70440492c80cfaea7d80323de" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced" is_pre_election: true
I20240502 21:29:44.910234 25260 raft_consensus.cc:2454] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 0.
I20240502 21:29:44.910252 25123 raft_consensus.cc:2454] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 0.
I20240502 21:29:44.910684 24790 leader_election.cc:302] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:44.910981 26007 raft_consensus.cc:2781] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:44.911084 26007 raft_consensus.cc:482] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:44.911199 26007 raft_consensus.cc:3037] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:44.913712 26007 raft_consensus.cc:504] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:44.914928 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf184a70440492c80cfaea7d80323de" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:44.914952 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf184a70440492c80cfaea7d80323de" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:44.915158 25260 raft_consensus.cc:3037] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:44.915176 25123 raft_consensus.cc:3037] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:44.918015 25123 raft_consensus.cc:2454] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 1.
I20240502 21:29:44.918015 25260 raft_consensus.cc:2454] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 1.
I20240502 21:29:44.914510 26007 leader_election.cc:288] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 1 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:44.918434 24790 leader_election.cc:302] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:44.918802 26250 raft_consensus.cc:2781] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:44.919669 26250 raft_consensus.cc:686] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [term 1 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:44.920059 26250 consensus_queue.cc:237] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:44.922581 24705 catalog_manager.cc:5484] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 0 to 1, leader changed from <none> to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 1 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } }
I20240502 21:29:45.144028 26200 raft_consensus.cc:482] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:45.144243 26200 raft_consensus.cc:504] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:45.144958 26200 leader_election.cc:288] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:45.145537 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175511733b714d2ea3cf877a0e5df418" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced" is_pre_election: true
I20240502 21:29:45.145637 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175511733b714d2ea3cf877a0e5df418" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b" is_pre_election: true
I20240502 21:29:45.145812 25123 raft_consensus.cc:2454] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 0.
I20240502 21:29:45.145882 24857 raft_consensus.cc:2454] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 0.
I20240502 21:29:45.146278 24926 leader_election.cc:302] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:45.146569 26200 raft_consensus.cc:2781] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:45.146696 26200 raft_consensus.cc:482] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:45.146802 26200 raft_consensus.cc:3037] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:45.148778 26200 raft_consensus.cc:504] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:45.149415 26200 leader_election.cc:288] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 1 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:45.149838 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175511733b714d2ea3cf877a0e5df418" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:45.150085 25123 raft_consensus.cc:3037] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:45.150015 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175511733b714d2ea3cf877a0e5df418" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:45.150238 24857 raft_consensus.cc:3037] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:45.152884 25123 raft_consensus.cc:2454] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 1.
I20240502 21:29:45.152915 24857 raft_consensus.cc:2454] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 1.
I20240502 21:29:45.153301 24926 leader_election.cc:302] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:45.153544 26200 raft_consensus.cc:2781] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:45.154071 26200 raft_consensus.cc:686] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:45.154460 26200 consensus_queue.cc:237] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:45.157073 24705 catalog_manager.cc:5484] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:45.286739 26224 consensus_queue.cc:1035] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:45.293761 26224 consensus_queue.cc:1035] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:45.491659 26250 consensus_queue.cc:1035] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:45.499962 26250 consensus_queue.cc:1035] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:29:45.542459 26256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:45.542846 26256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 21:29:45.543980 26200 consensus_queue.cc:1035] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:45.549674 26253 consensus_queue.cc:1035] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:29:45.553949 26256 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 213.404 ms
time per row: 0.106702 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:29:46.870968 24711 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53668:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:29:46.880295 24970 tablet_service.cc:1467] Processing CreateTablet for tablet 69df123d92d74693b4654c6a809bbe89 (DEFAULT_TABLE table=rebalance_test_table_5 [id=05d9656c29c746eda99f530f75a34b2a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:46.880420 25239 tablet_service.cc:1467] Processing CreateTablet for tablet 69df123d92d74693b4654c6a809bbe89 (DEFAULT_TABLE table=rebalance_test_table_5 [id=05d9656c29c746eda99f530f75a34b2a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:46.880811 24970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69df123d92d74693b4654c6a809bbe89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:46.880811 25239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69df123d92d74693b4654c6a809bbe89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:46.881675 24971 tablet_service.cc:1467] Processing CreateTablet for tablet 89c4eee02b444dc188172fd8c7b411bb (DEFAULT_TABLE table=rebalance_test_table_5 [id=05d9656c29c746eda99f530f75a34b2a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:46.882143 24971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89c4eee02b444dc188172fd8c7b411bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:46.882550 25102 tablet_service.cc:1467] Processing CreateTablet for tablet 3e32a87a85274194916bef376c7e558e (DEFAULT_TABLE table=rebalance_test_table_5 [id=05d9656c29c746eda99f530f75a34b2a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:46.883772 24837 tablet_service.cc:1467] Processing CreateTablet for tablet 3e32a87a85274194916bef376c7e558e (DEFAULT_TABLE table=rebalance_test_table_5 [id=05d9656c29c746eda99f530f75a34b2a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:46.884405 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e32a87a85274194916bef376c7e558e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:46.884634 24969 tablet_service.cc:1467] Processing CreateTablet for tablet 3e32a87a85274194916bef376c7e558e (DEFAULT_TABLE table=rebalance_test_table_5 [id=05d9656c29c746eda99f530f75a34b2a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:46.885043 24969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e32a87a85274194916bef376c7e558e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:46.887842 25103 tablet_service.cc:1467] Processing CreateTablet for tablet 69df123d92d74693b4654c6a809bbe89 (DEFAULT_TABLE table=rebalance_test_table_5 [id=05d9656c29c746eda99f530f75a34b2a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:46.888223 25102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e32a87a85274194916bef376c7e558e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:46.888774 25104 tablet_service.cc:1467] Processing CreateTablet for tablet 89c4eee02b444dc188172fd8c7b411bb (DEFAULT_TABLE table=rebalance_test_table_5 [id=05d9656c29c746eda99f530f75a34b2a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:46.889228 25104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89c4eee02b444dc188172fd8c7b411bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:46.892518 25240 tablet_service.cc:1467] Processing CreateTablet for tablet 89c4eee02b444dc188172fd8c7b411bb (DEFAULT_TABLE table=rebalance_test_table_5 [id=05d9656c29c746eda99f530f75a34b2a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:46.893064 25240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89c4eee02b444dc188172fd8c7b411bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:46.896211 25103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69df123d92d74693b4654c6a809bbe89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:46.903343 26323 tablet_bootstrap.cc:492] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced: Bootstrap starting.
I20240502 21:29:46.903309 26322 tablet_bootstrap.cc:492] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b: Bootstrap starting.
I20240502 21:29:46.905970 26323 tablet_bootstrap.cc:654] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:46.906445 26322 tablet_bootstrap.cc:654] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:46.907677 26320 tablet_bootstrap.cc:492] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c: Bootstrap starting.
I20240502 21:29:46.909662 26320 tablet_bootstrap.cc:654] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:46.914821 26320 tablet_bootstrap.cc:492] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c: No bootstrap required, opened a new log
I20240502 21:29:46.916141 26321 tablet_bootstrap.cc:492] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557: Bootstrap starting.
I20240502 21:29:46.915920 26320 ts_tablet_manager.cc:1367] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240502 21:29:46.917197 26323 tablet_bootstrap.cc:492] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced: No bootstrap required, opened a new log
I20240502 21:29:46.917344 26322 tablet_bootstrap.cc:492] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b: No bootstrap required, opened a new log
I20240502 21:29:46.917356 26323 ts_tablet_manager.cc:1367] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20240502 21:29:46.917606 26322 ts_tablet_manager.cc:1367] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20240502 21:29:46.917716 26321 tablet_bootstrap.cc:654] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:46.918177 26323 raft_consensus.cc:348] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:46.918176 26320 raft_consensus.cc:348] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:46.918399 26320 raft_consensus.cc:374] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:46.918418 26323 raft_consensus.cc:374] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:46.918521 26323 raft_consensus.cc:729] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Initialized, Role: FOLLOWER
I20240502 21:29:46.918550 26320 raft_consensus.cc:729] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Initialized, Role: FOLLOWER
I20240502 21:29:46.918746 26322 raft_consensus.cc:348] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:46.918884 26323 consensus_queue.cc:260] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:46.918926 26320 consensus_queue.cc:260] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:46.919134 26322 raft_consensus.cc:374] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:46.919312 26322 raft_consensus.cc:729] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Initialized, Role: FOLLOWER
I20240502 21:29:46.919649 26322 consensus_queue.cc:260] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:46.920421 26322 ts_tablet_manager.cc:1398] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:29:46.922307 26321 tablet_bootstrap.cc:492] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557: No bootstrap required, opened a new log
I20240502 21:29:46.922518 26321 ts_tablet_manager.cc:1367] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240502 21:29:46.923274 26320 ts_tablet_manager.cc:1398] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 21:29:46.923300 26323 ts_tablet_manager.cc:1398] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 21:29:46.923647 26320 tablet_bootstrap.cc:492] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c: Bootstrap starting.
I20240502 21:29:46.923805 26321 raft_consensus.cc:348] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:46.924079 26321 raft_consensus.cc:374] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:46.924218 26321 raft_consensus.cc:729] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Initialized, Role: FOLLOWER
I20240502 21:29:46.924525 26321 consensus_queue.cc:260] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:46.925647 26321 ts_tablet_manager.cc:1398] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:29:46.926015 26321 tablet_bootstrap.cc:492] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557: Bootstrap starting.
I20240502 21:29:46.926013 26320 tablet_bootstrap.cc:654] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:46.927999 26321 tablet_bootstrap.cc:654] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:46.928529 26323 tablet_bootstrap.cc:492] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced: Bootstrap starting.
I20240502 21:29:46.930840 26323 tablet_bootstrap.cc:654] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:46.930986 26320 tablet_bootstrap.cc:492] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c: No bootstrap required, opened a new log
I20240502 21:29:46.931222 26320 ts_tablet_manager.cc:1367] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20240502 21:29:46.932269 26320 raft_consensus.cc:348] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:46.932602 26320 raft_consensus.cc:374] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:46.932735 26320 raft_consensus.cc:729] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Initialized, Role: FOLLOWER
I20240502 21:29:46.933063 26320 consensus_queue.cc:260] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:46.933763 26323 tablet_bootstrap.cc:492] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced: No bootstrap required, opened a new log
I20240502 21:29:46.933905 26320 ts_tablet_manager.cc:1398] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:29:46.933965 26323 ts_tablet_manager.cc:1367] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20240502 21:29:46.934427 26321 tablet_bootstrap.cc:492] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557: No bootstrap required, opened a new log
I20240502 21:29:46.934587 26321 ts_tablet_manager.cc:1367] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20240502 21:29:46.934899 26323 raft_consensus.cc:348] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:46.935134 26323 raft_consensus.cc:374] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:46.935272 26323 raft_consensus.cc:729] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Initialized, Role: FOLLOWER
I20240502 21:29:46.935390 26321 raft_consensus.cc:348] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:46.935638 26321 raft_consensus.cc:374] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:46.935520 26323 consensus_queue.cc:260] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:46.935757 26321 raft_consensus.cc:729] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Initialized, Role: FOLLOWER
I20240502 21:29:46.936138 26323 ts_tablet_manager.cc:1398] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:29:46.936524 26323 tablet_bootstrap.cc:492] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced: Bootstrap starting.
I20240502 21:29:46.936815 26321 consensus_queue.cc:260] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:46.937482 26321 ts_tablet_manager.cc:1398] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:29:46.937980 26321 tablet_bootstrap.cc:492] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557: Bootstrap starting.
I20240502 21:29:46.938668 26323 tablet_bootstrap.cc:654] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:46.939635 26321 tablet_bootstrap.cc:654] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:46.941850 26321 tablet_bootstrap.cc:492] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557: No bootstrap required, opened a new log
I20240502 21:29:46.942013 26321 ts_tablet_manager.cc:1367] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240502 21:29:46.942247 26323 tablet_bootstrap.cc:492] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced: No bootstrap required, opened a new log
I20240502 21:29:46.942378 26323 ts_tablet_manager.cc:1367] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20240502 21:29:46.942799 26321 raft_consensus.cc:348] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:46.942912 26323 raft_consensus.cc:348] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:46.943017 26321 raft_consensus.cc:374] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:46.943106 26323 raft_consensus.cc:374] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:46.943122 26321 raft_consensus.cc:729] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Initialized, Role: FOLLOWER
I20240502 21:29:46.943205 26323 raft_consensus.cc:729] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Initialized, Role: FOLLOWER
I20240502 21:29:46.943326 26321 consensus_queue.cc:260] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:46.943431 26323 consensus_queue.cc:260] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:46.943920 26321 ts_tablet_manager.cc:1398] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:29:46.944985 26323 ts_tablet_manager.cc:1398] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:29:46.949429 26229 raft_consensus.cc:482] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:46.949543 26229 raft_consensus.cc:504] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:46.950059 26229 leader_election.cc:288] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:46.950496 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e32a87a85274194916bef376c7e558e" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b" is_pre_election: true
I20240502 21:29:46.950577 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e32a87a85274194916bef376c7e558e" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557" is_pre_election: true
I20240502 21:29:46.950706 24856 raft_consensus.cc:2454] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 0.
I20240502 21:29:46.950767 24991 raft_consensus.cc:2454] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 0.
I20240502 21:29:46.951203 25058 leader_election.cc:302] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:46.951454 26229 raft_consensus.cc:2781] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:46.951603 26229 raft_consensus.cc:482] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:46.951694 26229 raft_consensus.cc:3037] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:46.953747 26229 raft_consensus.cc:504] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:46.954236 26229 leader_election.cc:288] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:46.954631 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e32a87a85274194916bef376c7e558e" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:46.954713 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e32a87a85274194916bef376c7e558e" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:46.954806 24856 raft_consensus.cc:3037] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:46.954900 24991 raft_consensus.cc:3037] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:46.956961 24991 raft_consensus.cc:2454] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 1.
I20240502 21:29:46.956971 24856 raft_consensus.cc:2454] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 1.
I20240502 21:29:46.957330 25058 leader_election.cc:302] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:46.957628 26229 raft_consensus.cc:2781] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:46.958230 26229 raft_consensus.cc:686] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [term 1 LEADER]: Becoming Leader. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:46.958504 26229 consensus_queue.cc:237] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:46.961032 24711 catalog_manager.cc:5484] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bf90e514781418da921c046aa5d9ced (127.29.82.67). New cstate: current_term: 1 leader_uuid: "5bf90e514781418da921c046aa5d9ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:47.016381 26229 raft_consensus.cc:482] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:47.016641 26229 raft_consensus.cc:504] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:47.017416 26229 leader_election.cc:288] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:47.018177 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69df123d92d74693b4654c6a809bbe89" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557" is_pre_election: true
I20240502 21:29:47.018174 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69df123d92d74693b4654c6a809bbe89" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b7d5b9696b4046b522c880c748fe6c" is_pre_election: true
I20240502 21:29:47.018453 24991 raft_consensus.cc:2454] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 0.
I20240502 21:29:47.018443 25260 raft_consensus.cc:2454] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 0.
I20240502 21:29:47.018945 25058 leader_election.cc:302] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:47.019809 26229 raft_consensus.cc:2781] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:47.019909 26229 raft_consensus.cc:482] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:47.020020 26229 raft_consensus.cc:3037] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:47.028977 26260 raft_consensus.cc:482] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:47.029161 26260 raft_consensus.cc:504] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:47.029913 26260 leader_election.cc:288] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:47.030546 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c4eee02b444dc188172fd8c7b411bb" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced" is_pre_election: true
I20240502 21:29:47.030578 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c4eee02b444dc188172fd8c7b411bb" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557" is_pre_election: true
I20240502 21:29:47.030784 25123 raft_consensus.cc:2454] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 0.
I20240502 21:29:47.030822 24991 raft_consensus.cc:2454] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 0.
I20240502 21:29:47.031394 25191 leader_election.cc:302] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced; no voters:
I20240502 21:29:47.031831 26260 raft_consensus.cc:2781] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:47.031988 26260 raft_consensus.cc:482] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:47.032106 26260 raft_consensus.cc:3037] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:47.034960 26260 raft_consensus.cc:504] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:47.035543 26260 leader_election.cc:288] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:47.036015 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c4eee02b444dc188172fd8c7b411bb" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:47.036054 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c4eee02b444dc188172fd8c7b411bb" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:47.036190 25123 raft_consensus.cc:3037] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:47.036252 24991 raft_consensus.cc:3037] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:47.038105 24991 raft_consensus.cc:2454] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 1.
I20240502 21:29:47.038512 25191 leader_election.cc:302] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:47.038774 26260 raft_consensus.cc:2781] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:47.038933 26260 raft_consensus.cc:686] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:47.039211 26260 consensus_queue.cc:237] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:47.042666 26229 raft_consensus.cc:504] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:47.042843 25123 raft_consensus.cc:2454] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 1.
I20240502 21:29:47.043972 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69df123d92d74693b4654c6a809bbe89" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:47.044113 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69df123d92d74693b4654c6a809bbe89" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:47.044166 26229 leader_election.cc:288] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:47.044227 24991 raft_consensus.cc:3037] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:47.044327 25260 raft_consensus.cc:3037] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:47.045432 24711 catalog_manager.cc:5484] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 1 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:47.046101 24991 raft_consensus.cc:2454] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 1.
I20240502 21:29:47.046474 25058 leader_election.cc:302] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:47.046748 26229 raft_consensus.cc:2781] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:47.046806 25260 raft_consensus.cc:2454] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 1.
I20240502 21:29:47.046895 26229 raft_consensus.cc:686] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [term 1 LEADER]: Becoming Leader. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:47.047166 26229 consensus_queue.cc:237] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:47.050000 24708 catalog_manager.cc:5484] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bf90e514781418da921c046aa5d9ced (127.29.82.67). New cstate: current_term: 1 leader_uuid: "5bf90e514781418da921c046aa5d9ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:47.367146 26229 consensus_queue.cc:1035] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:47.374630 26229 consensus_queue.cc:1035] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:29:47.429870 26339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:47.430168 26339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:47.438400 26339 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 21:29:47.486174 26260 consensus_queue.cc:1035] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:47.521876 26260 consensus_queue.cc:1035] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:47.542141 26229 consensus_queue.cc:1035] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:47.547892 26334 consensus_queue.cc:1035] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 225.604 ms
time per row: 0.112802 ms
I20240502 21:29:47.822530 24708 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53668:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:29:47.830994 25240 tablet_service.cc:1467] Processing CreateTablet for tablet 986c305824414299ac30f1e50f514c8a (DEFAULT_TABLE table=rebalance_test_table_6 [id=d860b9155d354398b0a33659a73fc61e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:47.831527 25240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 986c305824414299ac30f1e50f514c8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:47.831425 24837 tablet_service.cc:1467] Processing CreateTablet for tablet 986c305824414299ac30f1e50f514c8a (DEFAULT_TABLE table=rebalance_test_table_6 [id=d860b9155d354398b0a33659a73fc61e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:47.831902 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 986c305824414299ac30f1e50f514c8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:47.832825 24970 tablet_service.cc:1467] Processing CreateTablet for tablet 986c305824414299ac30f1e50f514c8a (DEFAULT_TABLE table=rebalance_test_table_6 [id=d860b9155d354398b0a33659a73fc61e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:47.833274 24970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 986c305824414299ac30f1e50f514c8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:47.833302 24969 tablet_service.cc:1467] Processing CreateTablet for tablet 48e7c6534b544336ae216d3200f4f01f (DEFAULT_TABLE table=rebalance_test_table_6 [id=d860b9155d354398b0a33659a73fc61e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:47.833669 24969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e7c6534b544336ae216d3200f4f01f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:47.835988 25104 tablet_service.cc:1467] Processing CreateTablet for tablet 89b06cd6b69d41d0b49ad40f605a3e16 (DEFAULT_TABLE table=rebalance_test_table_6 [id=d860b9155d354398b0a33659a73fc61e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:47.836146 24836 tablet_service.cc:1467] Processing CreateTablet for tablet 89b06cd6b69d41d0b49ad40f605a3e16 (DEFAULT_TABLE table=rebalance_test_table_6 [id=d860b9155d354398b0a33659a73fc61e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:47.836269 24835 tablet_service.cc:1467] Processing CreateTablet for tablet 48e7c6534b544336ae216d3200f4f01f (DEFAULT_TABLE table=rebalance_test_table_6 [id=d860b9155d354398b0a33659a73fc61e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:47.836431 25104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89b06cd6b69d41d0b49ad40f605a3e16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:47.836508 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89b06cd6b69d41d0b49ad40f605a3e16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:47.837190 24835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e7c6534b544336ae216d3200f4f01f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:47.844126 25239 tablet_service.cc:1467] Processing CreateTablet for tablet 89b06cd6b69d41d0b49ad40f605a3e16 (DEFAULT_TABLE table=rebalance_test_table_6 [id=d860b9155d354398b0a33659a73fc61e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:47.844602 25239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89b06cd6b69d41d0b49ad40f605a3e16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:47.851107 25241 tablet_service.cc:1467] Processing CreateTablet for tablet 48e7c6534b544336ae216d3200f4f01f (DEFAULT_TABLE table=rebalance_test_table_6 [id=d860b9155d354398b0a33659a73fc61e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:47.851560 25241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e7c6534b544336ae216d3200f4f01f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:47.852463 26404 tablet_bootstrap.cc:492] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c: Bootstrap starting.
I20240502 21:29:47.855943 26402 tablet_bootstrap.cc:492] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b: Bootstrap starting.
I20240502 21:29:47.855984 26404 tablet_bootstrap.cc:654] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:47.856305 26400 tablet_bootstrap.cc:492] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557: Bootstrap starting.
I20240502 21:29:47.856696 26403 tablet_bootstrap.cc:492] T 89b06cd6b69d41d0b49ad40f605a3e16 P 5bf90e514781418da921c046aa5d9ced: Bootstrap starting.
I20240502 21:29:47.858201 26400 tablet_bootstrap.cc:654] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:47.860371 26403 tablet_bootstrap.cc:654] T 89b06cd6b69d41d0b49ad40f605a3e16 P 5bf90e514781418da921c046aa5d9ced: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:47.861445 26402 tablet_bootstrap.cc:654] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:47.861865 26404 tablet_bootstrap.cc:492] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c: No bootstrap required, opened a new log
I20240502 21:29:47.862068 26404 ts_tablet_manager.cc:1367] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240502 21:29:47.863054 26404 raft_consensus.cc:348] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:47.863449 26404 raft_consensus.cc:374] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:47.863622 26404 raft_consensus.cc:729] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Initialized, Role: FOLLOWER
I20240502 21:29:47.864024 26404 consensus_queue.cc:260] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:47.865041 26404 ts_tablet_manager.cc:1398] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:29:47.865653 26404 tablet_bootstrap.cc:492] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c: Bootstrap starting.
I20240502 21:29:47.867808 26404 tablet_bootstrap.cc:654] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:47.870927 26400 tablet_bootstrap.cc:492] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557: No bootstrap required, opened a new log
I20240502 21:29:47.871263 26400 ts_tablet_manager.cc:1367] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.004s
I20240502 21:29:47.871587 26402 tablet_bootstrap.cc:492] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b: No bootstrap required, opened a new log
I20240502 21:29:47.871735 26402 ts_tablet_manager.cc:1367] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.006s
I20240502 21:29:47.872130 26400 raft_consensus.cc:348] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:47.872327 26400 raft_consensus.cc:374] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:47.872426 26400 raft_consensus.cc:729] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Initialized, Role: FOLLOWER
I20240502 21:29:47.872534 26402 raft_consensus.cc:348] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:47.872732 26402 raft_consensus.cc:374] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:47.872666 26400 consensus_queue.cc:260] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:47.872833 26402 raft_consensus.cc:729] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Initialized, Role: FOLLOWER
I20240502 21:29:47.873037 26402 consensus_queue.cc:260] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:47.873680 26402 ts_tablet_manager.cc:1398] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:29:47.873920 26400 ts_tablet_manager.cc:1398] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:29:47.873975 26402 tablet_bootstrap.cc:492] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b: Bootstrap starting.
I20240502 21:29:47.874183 26400 tablet_bootstrap.cc:492] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557: Bootstrap starting.
I20240502 21:29:47.875648 26402 tablet_bootstrap.cc:654] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:47.875783 26400 tablet_bootstrap.cc:654] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:47.877923 26403 tablet_bootstrap.cc:492] T 89b06cd6b69d41d0b49ad40f605a3e16 P 5bf90e514781418da921c046aa5d9ced: No bootstrap required, opened a new log
I20240502 21:29:47.878088 26403 ts_tablet_manager.cc:1367] T 89b06cd6b69d41d0b49ad40f605a3e16 P 5bf90e514781418da921c046aa5d9ced: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.000s
I20240502 21:29:47.879107 26403 raft_consensus.cc:348] T 89b06cd6b69d41d0b49ad40f605a3e16 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:47.879298 26403 raft_consensus.cc:374] T 89b06cd6b69d41d0b49ad40f605a3e16 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:47.879390 26403 raft_consensus.cc:729] T 89b06cd6b69d41d0b49ad40f605a3e16 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Initialized, Role: FOLLOWER
I20240502 21:29:47.879635 26403 consensus_queue.cc:260] T 89b06cd6b69d41d0b49ad40f605a3e16 P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:47.880391 26403 ts_tablet_manager.cc:1398] T 89b06cd6b69d41d0b49ad40f605a3e16 P 5bf90e514781418da921c046aa5d9ced: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:29:47.884577 26404 tablet_bootstrap.cc:492] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c: No bootstrap required, opened a new log
I20240502 21:29:47.885169 26404 ts_tablet_manager.cc:1367] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.011s
I20240502 21:29:47.886075 26404 raft_consensus.cc:348] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:47.886360 26404 raft_consensus.cc:374] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:47.886504 26404 raft_consensus.cc:729] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Initialized, Role: FOLLOWER
I20240502 21:29:47.886818 26404 consensus_queue.cc:260] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:47.887437 26404 ts_tablet_manager.cc:1398] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:29:47.887866 26404 tablet_bootstrap.cc:492] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c: Bootstrap starting.
I20240502 21:29:47.889520 26404 tablet_bootstrap.cc:654] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:47.897845 26404 tablet_bootstrap.cc:492] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c: No bootstrap required, opened a new log
I20240502 21:29:47.898031 26404 ts_tablet_manager.cc:1367] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20240502 21:29:47.898245 26400 tablet_bootstrap.cc:492] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557: No bootstrap required, opened a new log
I20240502 21:29:47.898425 26400 ts_tablet_manager.cc:1367] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.013s
I20240502 21:29:47.898543 26402 tablet_bootstrap.cc:492] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b: No bootstrap required, opened a new log
I20240502 21:29:47.898738 26402 ts_tablet_manager.cc:1367] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.017s
I20240502 21:29:47.898869 26404 raft_consensus.cc:348] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:47.899138 26404 raft_consensus.cc:374] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:47.899243 26404 raft_consensus.cc:729] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Initialized, Role: FOLLOWER
I20240502 21:29:47.899209 26400 raft_consensus.cc:348] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:47.899461 26400 raft_consensus.cc:374] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:47.899538 26400 raft_consensus.cc:729] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Initialized, Role: FOLLOWER
I20240502 21:29:47.899477 26404 consensus_queue.cc:260] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:47.899632 26402 raft_consensus.cc:348] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:47.899900 26402 raft_consensus.cc:374] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:47.899860 26400 consensus_queue.cc:260] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:47.899976 26402 raft_consensus.cc:729] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Initialized, Role: FOLLOWER
I20240502 21:29:47.900143 26404 ts_tablet_manager.cc:1398] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:29:47.900267 26402 consensus_queue.cc:260] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:47.901326 26400 ts_tablet_manager.cc:1398] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:29:47.901849 26402 ts_tablet_manager.cc:1398] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:29:47.902213 26402 tablet_bootstrap.cc:492] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b: Bootstrap starting.
I20240502 21:29:47.903916 26402 tablet_bootstrap.cc:654] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:47.905967 26402 tablet_bootstrap.cc:492] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b: No bootstrap required, opened a new log
I20240502 21:29:47.906127 26402 ts_tablet_manager.cc:1367] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:29:47.906953 26402 raft_consensus.cc:348] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:47.907171 26402 raft_consensus.cc:374] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:47.907272 26402 raft_consensus.cc:729] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Initialized, Role: FOLLOWER
I20240502 21:29:47.907474 26402 consensus_queue.cc:260] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:47.907938 26402 ts_tablet_manager.cc:1398] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:29:47.981588 26250 raft_consensus.cc:482] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:47.981829 26250 raft_consensus.cc:504] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:47.982576 26250 leader_election.cc:288] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:47.983112 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c305824414299ac30f1e50f514c8a" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b7d5b9696b4046b522c880c748fe6c" is_pre_election: true
I20240502 21:29:47.983119 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c305824414299ac30f1e50f514c8a" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557" is_pre_election: true
I20240502 21:29:47.983384 25260 raft_consensus.cc:2454] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 0.
I20240502 21:29:47.983424 24992 raft_consensus.cc:2454] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 0.
I20240502 21:29:47.983862 24790 leader_election.cc:302] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:47.984306 26250 raft_consensus.cc:2781] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:47.984401 26250 raft_consensus.cc:482] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:47.984493 26250 raft_consensus.cc:3037] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:47.987304 26250 raft_consensus.cc:504] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:47.987977 26250 leader_election.cc:288] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 1 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:47.988333 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c305824414299ac30f1e50f514c8a" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:47.988368 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c305824414299ac30f1e50f514c8a" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:47.988503 25260 raft_consensus.cc:3037] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:47.988555 24992 raft_consensus.cc:3037] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:47.990542 24992 raft_consensus.cc:2454] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 1.
I20240502 21:29:47.990845 25260 raft_consensus.cc:2454] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 1.
I20240502 21:29:47.990959 24789 leader_election.cc:302] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:47.991269 26250 raft_consensus.cc:2781] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:47.991717 26250 raft_consensus.cc:686] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 1 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:47.991995 26250 consensus_queue.cc:237] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:47.994884 24708 catalog_manager.cc:5484] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 0 to 1, leader changed from <none> to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 1 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:48.036948 26369 raft_consensus.cc:482] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:48.037163 26369 raft_consensus.cc:504] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:48.037863 26369 leader_election.cc:288] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:48.038417 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b06cd6b69d41d0b49ad40f605a3e16" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b" is_pre_election: true
I20240502 21:29:48.038455 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b06cd6b69d41d0b49ad40f605a3e16" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced" is_pre_election: true
I20240502 21:29:48.038686 24857 raft_consensus.cc:2454] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 0.
I20240502 21:29:48.038695 25123 raft_consensus.cc:2454] T 89b06cd6b69d41d0b49ad40f605a3e16 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 0.
I20240502 21:29:48.039157 25191 leader_election.cc:302] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:48.039446 26369 raft_consensus.cc:2781] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:48.039563 26369 raft_consensus.cc:482] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:48.039688 26369 raft_consensus.cc:3037] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:48.041729 26369 raft_consensus.cc:504] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:48.042335 26369 leader_election.cc:288] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:48.042757 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b06cd6b69d41d0b49ad40f605a3e16" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:48.042861 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b06cd6b69d41d0b49ad40f605a3e16" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:48.042990 24857 raft_consensus.cc:3037] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:48.043087 25123 raft_consensus.cc:3037] T 89b06cd6b69d41d0b49ad40f605a3e16 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:48.045907 25123 raft_consensus.cc:2454] T 89b06cd6b69d41d0b49ad40f605a3e16 P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 1.
I20240502 21:29:48.045913 24857 raft_consensus.cc:2454] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 1.
I20240502 21:29:48.046299 25191 leader_election.cc:302] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced; no voters:
I20240502 21:29:48.046622 26369 raft_consensus.cc:2781] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:48.046782 26369 raft_consensus.cc:686] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:48.047104 26369 consensus_queue.cc:237] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:48.050369 24708 catalog_manager.cc:5484] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 1 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:48.071673 26369 raft_consensus.cc:482] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:48.071847 26369 raft_consensus.cc:504] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:48.072415 26369 leader_election.cc:288] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:48.072883 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e7c6534b544336ae216d3200f4f01f" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557" is_pre_election: true
I20240502 21:29:48.072968 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e7c6534b544336ae216d3200f4f01f" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b" is_pre_election: true
I20240502 21:29:48.073134 24992 raft_consensus.cc:2454] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 0.
I20240502 21:29:48.073204 24857 raft_consensus.cc:2454] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 0.
I20240502 21:29:48.073539 25191 leader_election.cc:302] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:48.073810 26369 raft_consensus.cc:2781] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:48.073912 26369 raft_consensus.cc:482] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:48.074013 26369 raft_consensus.cc:3037] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:48.075845 26369 raft_consensus.cc:504] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:48.076458 26369 leader_election.cc:288] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:48.076797 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e7c6534b544336ae216d3200f4f01f" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:48.076999 24992 raft_consensus.cc:3037] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:48.077003 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e7c6534b544336ae216d3200f4f01f" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:48.077229 24857 raft_consensus.cc:3037] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:48.078866 24992 raft_consensus.cc:2454] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 1.
I20240502 21:29:48.079238 24857 raft_consensus.cc:2454] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 1.
I20240502 21:29:48.079242 25191 leader_election.cc:302] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:48.079526 26369 raft_consensus.cc:2781] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:48.079746 26369 raft_consensus.cc:686] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:48.080005 26369 consensus_queue.cc:237] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:48.082954 24708 catalog_manager.cc:5484] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 1 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } }
I20240502 21:29:48.387470 26250 consensus_queue.cc:1035] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:48.403617 26250 consensus_queue.cc:1035] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:29:48.484622 26419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:48.485008 26419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:48.492606 26419 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240502 21:29:48.532931 26369 consensus_queue.cc:1035] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:48.539911 26336 consensus_queue.cc:1035] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:48.632179 24857 raft_consensus.cc:1261] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Refusing update from remote peer 45b7d5b9696b4046b522c880c748fe6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:29:48.632829 24992 raft_consensus.cc:1261] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Refusing update from remote peer 45b7d5b9696b4046b522c880c748fe6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:29:48.633448 26336 consensus_queue.cc:1035] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:48.633754 26369 consensus_queue.cc:1035] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:48.712213 26424 mvcc.cc:205] Tried to move back new op lower bound from 7023351351829774336 to 7023351349577605120. Current Snapshot: MvccSnapshot[applied={T|T < 7023351351829774336}]
INSERT report
rows total: 2000
time total: 284.194 ms
time per row: 0.142097 ms
I20240502 21:29:48.927217 24703 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53668:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:29:48.936846 25104 tablet_service.cc:1467] Processing CreateTablet for tablet f62981fdfb2349bd9fbe0e85b77ae492 (DEFAULT_TABLE table=rebalance_test_table_7 [id=687ef9f7f4de48ffb9bcc93d41c9165c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:48.937386 25104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f62981fdfb2349bd9fbe0e85b77ae492. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:48.937674 25239 tablet_service.cc:1467] Processing CreateTablet for tablet 083a9351f8ee4a44a86cb2d42f1ec558 (DEFAULT_TABLE table=rebalance_test_table_7 [id=687ef9f7f4de48ffb9bcc93d41c9165c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:48.938128 25239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 083a9351f8ee4a44a86cb2d42f1ec558. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:48.938333 24836 tablet_service.cc:1467] Processing CreateTablet for tablet f62981fdfb2349bd9fbe0e85b77ae492 (DEFAULT_TABLE table=rebalance_test_table_7 [id=687ef9f7f4de48ffb9bcc93d41c9165c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:48.938756 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f62981fdfb2349bd9fbe0e85b77ae492. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:48.938771 25103 tablet_service.cc:1467] Processing CreateTablet for tablet 083a9351f8ee4a44a86cb2d42f1ec558 (DEFAULT_TABLE table=rebalance_test_table_7 [id=687ef9f7f4de48ffb9bcc93d41c9165c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:48.939170 25103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 083a9351f8ee4a44a86cb2d42f1ec558. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:48.940472 24835 tablet_service.cc:1467] Processing CreateTablet for tablet 083a9351f8ee4a44a86cb2d42f1ec558 (DEFAULT_TABLE table=rebalance_test_table_7 [id=687ef9f7f4de48ffb9bcc93d41c9165c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:48.940882 24835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 083a9351f8ee4a44a86cb2d42f1ec558. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:48.941071 25241 tablet_service.cc:1467] Processing CreateTablet for tablet d643c15afbec4933a0d0a145f17f703c (DEFAULT_TABLE table=rebalance_test_table_7 [id=687ef9f7f4de48ffb9bcc93d41c9165c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:48.941481 25241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d643c15afbec4933a0d0a145f17f703c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:48.944453 25102 tablet_service.cc:1467] Processing CreateTablet for tablet d643c15afbec4933a0d0a145f17f703c (DEFAULT_TABLE table=rebalance_test_table_7 [id=687ef9f7f4de48ffb9bcc93d41c9165c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:48.944923 25102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d643c15afbec4933a0d0a145f17f703c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:48.949432 24970 tablet_service.cc:1467] Processing CreateTablet for tablet f62981fdfb2349bd9fbe0e85b77ae492 (DEFAULT_TABLE table=rebalance_test_table_7 [id=687ef9f7f4de48ffb9bcc93d41c9165c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:48.949971 24970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f62981fdfb2349bd9fbe0e85b77ae492. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:48.952005 24969 tablet_service.cc:1467] Processing CreateTablet for tablet d643c15afbec4933a0d0a145f17f703c (DEFAULT_TABLE table=rebalance_test_table_7 [id=687ef9f7f4de48ffb9bcc93d41c9165c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:48.952237 26485 tablet_bootstrap.cc:492] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced: Bootstrap starting.
I20240502 21:29:48.952387 24969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d643c15afbec4933a0d0a145f17f703c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:48.954183 26485 tablet_bootstrap.cc:654] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:48.958925 26487 tablet_bootstrap.cc:492] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b: Bootstrap starting.
I20240502 21:29:48.961484 26487 tablet_bootstrap.cc:654] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:48.962850 26485 tablet_bootstrap.cc:492] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced: No bootstrap required, opened a new log
I20240502 21:29:48.963068 26485 ts_tablet_manager.cc:1367] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240502 21:29:48.965332 26487 tablet_bootstrap.cc:492] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b: No bootstrap required, opened a new log
I20240502 21:29:48.965478 26487 ts_tablet_manager.cc:1367] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 21:29:48.966167 26487 raft_consensus.cc:348] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:48.966361 26487 raft_consensus.cc:374] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:48.966468 26487 raft_consensus.cc:729] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Initialized, Role: FOLLOWER
I20240502 21:29:48.966763 26487 consensus_queue.cc:260] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:48.967397 26487 ts_tablet_manager.cc:1398] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:29:48.967763 26487 tablet_bootstrap.cc:492] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b: Bootstrap starting.
I20240502 21:29:48.969303 26487 tablet_bootstrap.cc:654] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:48.971143 26487 tablet_bootstrap.cc:492] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b: No bootstrap required, opened a new log
I20240502 21:29:48.971513 26487 ts_tablet_manager.cc:1367] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:29:48.972285 26487 raft_consensus.cc:348] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:48.972546 26487 raft_consensus.cc:374] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:48.972656 26487 raft_consensus.cc:729] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Initialized, Role: FOLLOWER
I20240502 21:29:48.972909 26487 consensus_queue.cc:260] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:48.973735 26487 ts_tablet_manager.cc:1398] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:29:48.980094 26485 raft_consensus.cc:348] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:48.980392 26485 raft_consensus.cc:374] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:48.980437 26491 tablet_bootstrap.cc:492] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557: Bootstrap starting.
I20240502 21:29:48.980495 26485 raft_consensus.cc:729] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Initialized, Role: FOLLOWER
I20240502 21:29:48.980739 26485 consensus_queue.cc:260] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:48.981365 26485 ts_tablet_manager.cc:1398] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240502 21:29:48.989995 26485 tablet_bootstrap.cc:492] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced: Bootstrap starting.
I20240502 21:29:48.991645 26485 tablet_bootstrap.cc:654] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:48.992767 26491 tablet_bootstrap.cc:654] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:48.999686 26486 tablet_bootstrap.cc:492] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c: Bootstrap starting.
I20240502 21:29:48.999814 26485 tablet_bootstrap.cc:492] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced: No bootstrap required, opened a new log
I20240502 21:29:48.999966 26485 ts_tablet_manager.cc:1367] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20240502 21:29:49.000624 26485 raft_consensus.cc:348] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:49.000816 26485 raft_consensus.cc:374] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:49.000896 26485 raft_consensus.cc:729] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Initialized, Role: FOLLOWER
I20240502 21:29:49.001092 26485 consensus_queue.cc:260] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:49.001595 26485 ts_tablet_manager.cc:1398] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:29:49.002410 26486 tablet_bootstrap.cc:654] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:49.004130 26334 raft_consensus.cc:482] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:49.004276 26334 raft_consensus.cc:504] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:49.004832 26334 leader_election.cc:288] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:49.005623 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62981fdfb2349bd9fbe0e85b77ae492" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557" is_pre_election: true
I20240502 21:29:49.005718 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62981fdfb2349bd9fbe0e85b77ae492" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b" is_pre_election: true
I20240502 21:29:49.006035 24857 raft_consensus.cc:2454] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 0.
I20240502 21:29:49.006548 25058 leader_election.cc:302] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b; no voters:
W20240502 21:29:49.007067 25058 leader_election.cc:341] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:29:49.007457 26491 tablet_bootstrap.cc:492] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557: No bootstrap required, opened a new log
I20240502 21:29:49.007711 26491 ts_tablet_manager.cc:1367] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.006s
I20240502 21:29:49.008554 26334 raft_consensus.cc:2781] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:49.008647 26334 raft_consensus.cc:482] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:49.008736 26334 raft_consensus.cc:3037] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:49.008889 26491 raft_consensus.cc:348] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:49.009158 26491 raft_consensus.cc:374] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:49.009261 26491 raft_consensus.cc:729] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Initialized, Role: FOLLOWER
I20240502 21:29:49.009624 26491 consensus_queue.cc:260] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:49.010488 26491 ts_tablet_manager.cc:1398] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:29:49.010799 26491 tablet_bootstrap.cc:492] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557: Bootstrap starting.
I20240502 21:29:49.011427 26486 tablet_bootstrap.cc:492] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c: No bootstrap required, opened a new log
I20240502 21:29:49.011408 26334 raft_consensus.cc:504] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:49.011606 26486 ts_tablet_manager.cc:1367] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20240502 21:29:49.012586 26486 raft_consensus.cc:348] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:49.012804 26486 raft_consensus.cc:374] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:49.012907 26486 raft_consensus.cc:729] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Initialized, Role: FOLLOWER
I20240502 21:29:49.013180 26486 consensus_queue.cc:260] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:49.013903 26486 ts_tablet_manager.cc:1398] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:29:49.014210 26486 tablet_bootstrap.cc:492] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c: Bootstrap starting.
I20240502 21:29:49.015944 26486 tablet_bootstrap.cc:654] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:49.023315 26485 tablet_bootstrap.cc:492] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced: Bootstrap starting.
I20240502 21:29:49.024780 26485 tablet_bootstrap.cc:654] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:49.024991 26491 tablet_bootstrap.cc:654] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:49.025924 26334 leader_election.cc:288] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:49.026403 26486 tablet_bootstrap.cc:492] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c: No bootstrap required, opened a new log
I20240502 21:29:49.026556 26486 ts_tablet_manager.cc:1367] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20240502 21:29:49.026681 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62981fdfb2349bd9fbe0e85b77ae492" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:49.026733 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62981fdfb2349bd9fbe0e85b77ae492" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:49.026957 24857 raft_consensus.cc:3037] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:29:49.027344 25058 leader_election.cc:341] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:29:49.027372 26486 raft_consensus.cc:348] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:49.027629 26486 raft_consensus.cc:374] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:49.027707 26486 raft_consensus.cc:729] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Initialized, Role: FOLLOWER
I20240502 21:29:49.027961 26486 consensus_queue.cc:260] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:49.028550 26486 ts_tablet_manager.cc:1398] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:29:49.030696 24857 raft_consensus.cc:2454] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 1.
I20240502 21:29:49.031204 26491 tablet_bootstrap.cc:492] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557: No bootstrap required, opened a new log
I20240502 21:29:49.031462 26491 ts_tablet_manager.cc:1367] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.002s
I20240502 21:29:49.031818 25058 leader_election.cc:302] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b; no voters: 64d97dd1674d43ccadd7b4a38eeff557
I20240502 21:29:49.032334 26491 raft_consensus.cc:348] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:49.032671 26491 raft_consensus.cc:374] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:49.032840 26491 raft_consensus.cc:729] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Initialized, Role: FOLLOWER
I20240502 21:29:49.033150 26491 consensus_queue.cc:260] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:49.033763 26334 raft_consensus.cc:2781] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:49.033838 26491 ts_tablet_manager.cc:1398] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:29:49.035121 26485 tablet_bootstrap.cc:492] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced: No bootstrap required, opened a new log
I20240502 21:29:49.035341 26485 ts_tablet_manager.cc:1367] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240502 21:29:49.036358 26485 raft_consensus.cc:348] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:49.036607 26485 raft_consensus.cc:374] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:49.036747 26485 raft_consensus.cc:729] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Initialized, Role: FOLLOWER
I20240502 21:29:49.037043 26485 consensus_queue.cc:260] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:49.039783 26369 raft_consensus.cc:482] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:49.039912 26369 raft_consensus.cc:504] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:49.040483 26369 leader_election.cc:288] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:49.041252 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "083a9351f8ee4a44a86cb2d42f1ec558" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced" is_pre_election: true
I20240502 21:29:49.041527 25123 raft_consensus.cc:2454] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 0.
I20240502 21:29:49.042492 25191 leader_election.cc:302] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced; no voters:
I20240502 21:29:49.042965 26369 raft_consensus.cc:2781] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:49.043123 26369 raft_consensus.cc:482] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:49.043223 26369 raft_consensus.cc:3037] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:49.048141 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "083a9351f8ee4a44a86cb2d42f1ec558" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b" is_pre_election: true
I20240502 21:29:49.048419 24857 raft_consensus.cc:2454] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 0.
I20240502 21:29:49.049669 26369 raft_consensus.cc:504] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:49.051126 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "083a9351f8ee4a44a86cb2d42f1ec558" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:49.051381 25123 raft_consensus.cc:3037] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:49.057519 25123 raft_consensus.cc:2454] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 1.
I20240502 21:29:49.058125 25191 leader_election.cc:302] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced; no voters:
I20240502 21:29:49.058549 26474 raft_consensus.cc:2781] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:49.058699 26474 raft_consensus.cc:686] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:49.059007 26474 consensus_queue.cc:237] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:49.059468 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "083a9351f8ee4a44a86cb2d42f1ec558" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:49.059767 24857 raft_consensus.cc:3037] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:49.066088 26369 leader_election.cc:288] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:49.066371 24857 raft_consensus.cc:2454] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 1.
I20240502 21:29:49.078586 26503 raft_consensus.cc:482] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:49.084116 26503 raft_consensus.cc:504] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:49.086042 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d643c15afbec4933a0d0a145f17f703c" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b7d5b9696b4046b522c880c748fe6c" is_pre_election: true
I20240502 21:29:49.086450 25260 raft_consensus.cc:2454] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 0.
I20240502 21:29:49.087191 25059 leader_election.cc:302] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced; no voters:
I20240502 21:29:49.088690 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d643c15afbec4933a0d0a145f17f703c" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557" is_pre_election: true
I20240502 21:29:49.089004 24992 raft_consensus.cc:2454] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 0.
I20240502 21:29:49.092638 26500 raft_consensus.cc:2781] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:49.094136 26500 raft_consensus.cc:482] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:49.094415 26500 raft_consensus.cc:3037] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:49.093037 26503 leader_election.cc:288] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:49.095429 24703 catalog_manager.cc:5484] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 1 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } }
I20240502 21:29:49.092877 26485 ts_tablet_manager.cc:1398] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced: Time spent starting tablet: real 0.057s user 0.012s sys 0.043s
I20240502 21:29:49.097565 26500 raft_consensus.cc:504] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:49.098564 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d643c15afbec4933a0d0a145f17f703c" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:49.098770 25260 raft_consensus.cc:3037] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:49.101553 25260 raft_consensus.cc:2454] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 1.
I20240502 21:29:49.102120 25059 leader_election.cc:302] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced; no voters:
I20240502 21:29:49.102603 26503 raft_consensus.cc:2781] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:49.102744 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d643c15afbec4933a0d0a145f17f703c" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:49.102980 24992 raft_consensus.cc:3037] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:49.103008 26503 raft_consensus.cc:686] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 1 LEADER]: Becoming Leader. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:49.103506 26503 consensus_queue.cc:237] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:49.105996 24992 raft_consensus.cc:2454] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 1.
I20240502 21:29:49.106909 26334 raft_consensus.cc:686] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [term 1 LEADER]: Becoming Leader. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:49.106962 26500 leader_election.cc:288] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:49.107365 26334 consensus_queue.cc:237] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:49.111090 24703 catalog_manager.cc:5484] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bf90e514781418da921c046aa5d9ced (127.29.82.67). New cstate: current_term: 1 leader_uuid: "5bf90e514781418da921c046aa5d9ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } }
I20240502 21:29:49.115995 24703 catalog_manager.cc:5484] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bf90e514781418da921c046aa5d9ced (127.29.82.67). New cstate: current_term: 1 leader_uuid: "5bf90e514781418da921c046aa5d9ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:49.274327 26315 raft_consensus.cc:482] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:49.274549 26315 raft_consensus.cc:504] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:49.275460 26315 leader_election.cc:288] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:49.275962 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62981fdfb2349bd9fbe0e85b77ae492" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced" is_pre_election: true
I20240502 21:29:49.276028 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62981fdfb2349bd9fbe0e85b77ae492" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b" is_pre_election: true
I20240502 21:29:49.276410 24857 raft_consensus.cc:2379] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 64d97dd1674d43ccadd7b4a38eeff557 in current term 1: Already voted for candidate 5bf90e514781418da921c046aa5d9ced in this term.
I20240502 21:29:49.276877 24926 leader_election.cc:302] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557; no voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b
I20240502 21:29:49.277163 26315 raft_consensus.cc:3037] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:49.279224 26315 raft_consensus.cc:2726] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 21:29:49.488224 26506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:49.488685 26506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:49.501258 26506 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:29:49.517092 26334 consensus_queue.cc:1035] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:49.524691 26334 consensus_queue.cc:1035] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_7'
I20240502 21:29:49.602967 26503 consensus_queue.cc:1035] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:49.630712 26474 consensus_queue.cc:1035] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:49.632053 26334 consensus_queue.cc:1035] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:49.646682 26369 consensus_queue.cc:1035] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 261.767 ms
time per row: 0.130884 ms
I20240502 21:29:49.933171 24708 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53668:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:29:49.941715 24836 tablet_service.cc:1467] Processing CreateTablet for tablet 3c1ce7cf99164a9a848cbb22ad2052ea (DEFAULT_TABLE table=rebalance_test_table_8 [id=450159e7a5d544628b224a736d287477]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:49.942188 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c1ce7cf99164a9a848cbb22ad2052ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:49.942222 24969 tablet_service.cc:1467] Processing CreateTablet for tablet 3c1ce7cf99164a9a848cbb22ad2052ea (DEFAULT_TABLE table=rebalance_test_table_8 [id=450159e7a5d544628b224a736d287477]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:49.942600 24969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c1ce7cf99164a9a848cbb22ad2052ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:49.943454 25239 tablet_service.cc:1467] Processing CreateTablet for tablet bdecfa03390e405ca4925f8d1a40e0ca (DEFAULT_TABLE table=rebalance_test_table_8 [id=450159e7a5d544628b224a736d287477]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:49.943841 25239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdecfa03390e405ca4925f8d1a40e0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:49.947669 25241 tablet_service.cc:1467] Processing CreateTablet for tablet 22e48679f46b4b72aa0f0736c76be6f1 (DEFAULT_TABLE table=rebalance_test_table_8 [id=450159e7a5d544628b224a736d287477]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:49.948016 25241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22e48679f46b4b72aa0f0736c76be6f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:49.948611 25102 tablet_service.cc:1467] Processing CreateTablet for tablet 3c1ce7cf99164a9a848cbb22ad2052ea (DEFAULT_TABLE table=rebalance_test_table_8 [id=450159e7a5d544628b224a736d287477]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:49.949028 25102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c1ce7cf99164a9a848cbb22ad2052ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:49.952862 24835 tablet_service.cc:1467] Processing CreateTablet for tablet bdecfa03390e405ca4925f8d1a40e0ca (DEFAULT_TABLE table=rebalance_test_table_8 [id=450159e7a5d544628b224a736d287477]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:49.953076 24970 tablet_service.cc:1467] Processing CreateTablet for tablet bdecfa03390e405ca4925f8d1a40e0ca (DEFAULT_TABLE table=rebalance_test_table_8 [id=450159e7a5d544628b224a736d287477]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:49.953267 24835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdecfa03390e405ca4925f8d1a40e0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:49.953544 24970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdecfa03390e405ca4925f8d1a40e0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:49.955960 24837 tablet_service.cc:1467] Processing CreateTablet for tablet 22e48679f46b4b72aa0f0736c76be6f1 (DEFAULT_TABLE table=rebalance_test_table_8 [id=450159e7a5d544628b224a736d287477]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:49.956099 25103 tablet_service.cc:1467] Processing CreateTablet for tablet 22e48679f46b4b72aa0f0736c76be6f1 (DEFAULT_TABLE table=rebalance_test_table_8 [id=450159e7a5d544628b224a736d287477]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:49.956308 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22e48679f46b4b72aa0f0736c76be6f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:49.956485 25103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22e48679f46b4b72aa0f0736c76be6f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:49.957002 26573 tablet_bootstrap.cc:492] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b: Bootstrap starting.
I20240502 21:29:49.958303 26572 tablet_bootstrap.cc:492] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c: Bootstrap starting.
I20240502 21:29:49.958909 26573 tablet_bootstrap.cc:654] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:49.959925 26572 tablet_bootstrap.cc:654] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:49.984252 26578 tablet_bootstrap.cc:492] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced: Bootstrap starting.
I20240502 21:29:49.985169 26576 tablet_bootstrap.cc:492] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557: Bootstrap starting.
I20240502 21:29:49.990046 26576 tablet_bootstrap.cc:654] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:49.990435 26578 tablet_bootstrap.cc:654] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:49.995887 26578 tablet_bootstrap.cc:492] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced: No bootstrap required, opened a new log
I20240502 21:29:49.996099 26578 ts_tablet_manager.cc:1367] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.002s
I20240502 21:29:49.996182 26576 tablet_bootstrap.cc:492] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557: No bootstrap required, opened a new log
I20240502 21:29:49.996340 26576 ts_tablet_manager.cc:1367] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240502 21:29:49.996824 26572 tablet_bootstrap.cc:492] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c: No bootstrap required, opened a new log
I20240502 21:29:49.997030 26572 ts_tablet_manager.cc:1367] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c: Time spent bootstrapping tablet: real 0.039s user 0.019s sys 0.010s
I20240502 21:29:49.997145 26576 raft_consensus.cc:348] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:49.997392 26576 raft_consensus.cc:374] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:49.997232 26578 raft_consensus.cc:348] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:49.997493 26576 raft_consensus.cc:729] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Initialized, Role: FOLLOWER
I20240502 21:29:49.997653 26578 raft_consensus.cc:374] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:49.997792 26578 raft_consensus.cc:729] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Initialized, Role: FOLLOWER
I20240502 21:29:49.997713 26576 consensus_queue.cc:260] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:49.998066 26572 raft_consensus.cc:348] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:49.998106 26578 consensus_queue.cc:260] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:49.998304 26572 raft_consensus.cc:374] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:49.998423 26572 raft_consensus.cc:729] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Initialized, Role: FOLLOWER
I20240502 21:29:49.998690 26572 consensus_queue.cc:260] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:49.999212 26572 ts_tablet_manager.cc:1398] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:29:49.999521 26572 tablet_bootstrap.cc:492] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c: Bootstrap starting.
I20240502 21:29:50.000584 26578 ts_tablet_manager.cc:1398] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 21:29:50.001042 26578 tablet_bootstrap.cc:492] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced: Bootstrap starting.
I20240502 21:29:50.001416 26572 tablet_bootstrap.cc:654] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:50.003032 26578 tablet_bootstrap.cc:654] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:50.008790 26578 tablet_bootstrap.cc:492] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced: No bootstrap required, opened a new log
I20240502 21:29:50.008944 26578 ts_tablet_manager.cc:1367] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240502 21:29:50.011536 26573 tablet_bootstrap.cc:492] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b: No bootstrap required, opened a new log
I20240502 21:29:50.011746 26573 ts_tablet_manager.cc:1367] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.032s
I20240502 21:29:50.012507 26573 raft_consensus.cc:348] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:50.012696 26573 raft_consensus.cc:374] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:50.012765 26573 raft_consensus.cc:729] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Initialized, Role: FOLLOWER
I20240502 21:29:50.012980 26573 consensus_queue.cc:260] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:50.013612 26573 ts_tablet_manager.cc:1398] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:29:50.013901 26573 tablet_bootstrap.cc:492] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b: Bootstrap starting.
I20240502 21:29:50.016001 26576 ts_tablet_manager.cc:1398] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20240502 21:29:50.016292 26576 tablet_bootstrap.cc:492] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557: Bootstrap starting.
I20240502 21:29:50.019423 26573 tablet_bootstrap.cc:654] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:50.011282 26578 raft_consensus.cc:348] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:50.020561 26576 tablet_bootstrap.cc:654] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:50.021009 26578 raft_consensus.cc:374] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:50.021132 26578 raft_consensus.cc:729] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Initialized, Role: FOLLOWER
I20240502 21:29:50.023654 26578 consensus_queue.cc:260] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:50.024351 26578 ts_tablet_manager.cc:1398] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240502 21:29:50.030423 26573 tablet_bootstrap.cc:492] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b: No bootstrap required, opened a new log
I20240502 21:29:50.030587 26573 ts_tablet_manager.cc:1367] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.004s
I20240502 21:29:50.031360 26572 tablet_bootstrap.cc:492] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c: No bootstrap required, opened a new log
I20240502 21:29:50.031334 26573 raft_consensus.cc:348] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:50.031543 26572 ts_tablet_manager.cc:1367] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.009s
I20240502 21:29:50.031553 26573 raft_consensus.cc:374] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:50.031657 26573 raft_consensus.cc:729] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Initialized, Role: FOLLOWER
I20240502 21:29:50.031920 26573 consensus_queue.cc:260] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:50.032559 26573 ts_tablet_manager.cc:1398] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:29:50.032441 26572 raft_consensus.cc:348] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:50.032670 26572 raft_consensus.cc:374] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:50.032742 26572 raft_consensus.cc:729] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Initialized, Role: FOLLOWER
I20240502 21:29:50.032971 26572 consensus_queue.cc:260] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:50.033547 26572 ts_tablet_manager.cc:1398] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:29:50.032877 26573 tablet_bootstrap.cc:492] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b: Bootstrap starting.
I20240502 21:29:50.039117 26576 tablet_bootstrap.cc:492] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557: No bootstrap required, opened a new log
I20240502 21:29:50.039284 26576 ts_tablet_manager.cc:1367] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.004s
I20240502 21:29:50.040036 26576 raft_consensus.cc:348] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:50.040287 26576 raft_consensus.cc:374] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:50.040369 26576 raft_consensus.cc:729] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Initialized, Role: FOLLOWER
I20240502 21:29:50.040620 26576 consensus_queue.cc:260] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:50.041285 26576 ts_tablet_manager.cc:1398] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:29:50.042016 26573 tablet_bootstrap.cc:654] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:50.048818 26573 tablet_bootstrap.cc:492] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b: No bootstrap required, opened a new log
I20240502 21:29:50.048972 26573 ts_tablet_manager.cc:1367] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20240502 21:29:50.049703 26573 raft_consensus.cc:348] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:50.049898 26573 raft_consensus.cc:374] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:50.049979 26573 raft_consensus.cc:729] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Initialized, Role: FOLLOWER
I20240502 21:29:50.050176 26573 consensus_queue.cc:260] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:50.050719 26573 ts_tablet_manager.cc:1398] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:29:50.122665 26503 raft_consensus.cc:482] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:50.122866 26503 raft_consensus.cc:504] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:50.123325 26535 raft_consensus.cc:482] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:50.123618 26503 leader_election.cc:288] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:50.123512 26535 raft_consensus.cc:504] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:50.124116 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1ce7cf99164a9a848cbb22ad2052ea" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b" is_pre_election: true
I20240502 21:29:50.124238 26535 leader_election.cc:288] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:50.124394 24856 raft_consensus.cc:2454] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 0.
I20240502 21:29:50.124521 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1ce7cf99164a9a848cbb22ad2052ea" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557" is_pre_election: true
I20240502 21:29:50.124758 24992 raft_consensus.cc:2454] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 0.
I20240502 21:29:50.124930 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1ce7cf99164a9a848cbb22ad2052ea" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b" is_pre_election: true
I20240502 21:29:50.125010 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1ce7cf99164a9a848cbb22ad2052ea" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced" is_pre_election: true
I20240502 21:29:50.125188 24856 raft_consensus.cc:2454] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 0.
I20240502 21:29:50.125224 25123 raft_consensus.cc:2454] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 0.
I20240502 21:29:50.125294 25058 leader_election.cc:302] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:50.125625 26503 raft_consensus.cc:2781] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:50.125582 24926 leader_election.cc:302] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:50.125722 26503 raft_consensus.cc:482] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:50.125834 26503 raft_consensus.cc:3037] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:50.125921 26535 raft_consensus.cc:2781] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:50.126045 26535 raft_consensus.cc:482] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:50.126140 26535 raft_consensus.cc:3037] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:50.127799 26503 raft_consensus.cc:504] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:50.127990 26535 raft_consensus.cc:504] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:50.128315 26503 leader_election.cc:288] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:50.128633 26535 leader_election.cc:288] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 1 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:50.128742 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1ce7cf99164a9a848cbb22ad2052ea" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:50.128902 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1ce7cf99164a9a848cbb22ad2052ea" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:50.128986 24856 raft_consensus.cc:3037] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:50.129096 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1ce7cf99164a9a848cbb22ad2052ea" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:50.129194 24992 raft_consensus.cc:2379] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5bf90e514781418da921c046aa5d9ced in current term 1: Already voted for candidate 64d97dd1674d43ccadd7b4a38eeff557 in this term.
I20240502 21:29:50.129774 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1ce7cf99164a9a848cbb22ad2052ea" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:50.130041 25123 raft_consensus.cc:2379] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 64d97dd1674d43ccadd7b4a38eeff557 in current term 1: Already voted for candidate 5bf90e514781418da921c046aa5d9ced in this term.
I20240502 21:29:50.130417 24926 leader_election.cc:302] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557; no voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b
I20240502 21:29:50.130678 26535 raft_consensus.cc:2726] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 21:29:50.131319 24856 raft_consensus.cc:2454] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 1.
I20240502 21:29:50.131709 25058 leader_election.cc:302] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b; no voters: 64d97dd1674d43ccadd7b4a38eeff557
I20240502 21:29:50.131961 26503 raft_consensus.cc:2781] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:50.132122 26503 raft_consensus.cc:686] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 1 LEADER]: Becoming Leader. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:50.132359 26503 consensus_queue.cc:237] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:50.135246 24710 catalog_manager.cc:5484] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bf90e514781418da921c046aa5d9ced (127.29.82.67). New cstate: current_term: 1 leader_uuid: "5bf90e514781418da921c046aa5d9ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:50.198030 26369 raft_consensus.cc:482] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:50.198261 26369 raft_consensus.cc:504] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:50.198966 26369 leader_election.cc:288] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:50.199628 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdecfa03390e405ca4925f8d1a40e0ca" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557" is_pre_election: true
I20240502 21:29:50.199680 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdecfa03390e405ca4925f8d1a40e0ca" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b" is_pre_election: true
I20240502 21:29:50.199906 24992 raft_consensus.cc:2454] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 0.
I20240502 21:29:50.199927 24856 raft_consensus.cc:2454] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 0.
I20240502 21:29:50.200372 25191 leader_election.cc:302] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:50.200668 26369 raft_consensus.cc:2781] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:50.200757 26369 raft_consensus.cc:482] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:50.200867 26369 raft_consensus.cc:3037] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:50.202796 26369 raft_consensus.cc:504] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:50.203343 26369 leader_election.cc:288] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:50.203853 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdecfa03390e405ca4925f8d1a40e0ca" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:50.203931 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdecfa03390e405ca4925f8d1a40e0ca" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:50.204077 24992 raft_consensus.cc:3037] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:50.204138 24856 raft_consensus.cc:3037] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:50.206054 24856 raft_consensus.cc:2454] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 1.
I20240502 21:29:50.206115 24992 raft_consensus.cc:2454] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 1.
I20240502 21:29:50.206467 25191 leader_election.cc:302] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:50.206739 26369 raft_consensus.cc:2781] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:50.206909 26369 raft_consensus.cc:686] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:50.207192 26369 consensus_queue.cc:237] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:50.210503 24710 catalog_manager.cc:5484] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 1 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } }
I20240502 21:29:50.366326 26415 raft_consensus.cc:482] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:50.366525 26415 raft_consensus.cc:504] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:50.367179 26415 leader_election.cc:288] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:50.367616 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22e48679f46b4b72aa0f0736c76be6f1" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b7d5b9696b4046b522c880c748fe6c" is_pre_election: true
I20240502 21:29:50.367708 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22e48679f46b4b72aa0f0736c76be6f1" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced" is_pre_election: true
I20240502 21:29:50.367861 25260 raft_consensus.cc:2454] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 0.
I20240502 21:29:50.367923 25123 raft_consensus.cc:2454] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 0.
I20240502 21:29:50.368283 24790 leader_election.cc:302] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:50.368546 26415 raft_consensus.cc:2781] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:50.368675 26415 raft_consensus.cc:482] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:50.368772 26415 raft_consensus.cc:3037] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:50.370656 26415 raft_consensus.cc:504] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:50.371208 26415 leader_election.cc:288] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 1 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:50.371634 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22e48679f46b4b72aa0f0736c76be6f1" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:50.371855 25260 raft_consensus.cc:3037] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:50.371795 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22e48679f46b4b72aa0f0736c76be6f1" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:50.372018 25123 raft_consensus.cc:3037] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:50.373965 25260 raft_consensus.cc:2454] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 1.
I20240502 21:29:50.374193 25123 raft_consensus.cc:2454] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 1.
I20240502 21:29:50.374385 24790 leader_election.cc:302] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:50.374686 26415 raft_consensus.cc:2781] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:50.374836 26415 raft_consensus.cc:686] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [term 1 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:50.375106 26415 consensus_queue.cc:237] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:50.378037 24710 catalog_manager.cc:5484] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 0 to 1, leader changed from <none> to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 1 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:50.524572 26503 consensus_queue.cc:1035] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:50.530155 26503 consensus_queue.cc:1035] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:50.716204 26474 consensus_queue.cc:1035] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:50.722664 26474 consensus_queue.cc:1035] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:29:50.767664 26589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:50.768024 26589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:50.775550 26589 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:29:50.798970 26415 consensus_queue.cc:1035] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:50.806054 26556 consensus_queue.cc:1035] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_8'
W20240502 21:29:51.072182 24926 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.82.65:33021, user_credentials={real_user=slave}} blocked reactor thread for 47818.4us
INSERT report
rows total: 2000
time total: 284.066 ms
time per row: 0.142033 ms
I20240502 21:29:51.208112 24703 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53668:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:29:51.217265 25103 tablet_service.cc:1467] Processing CreateTablet for tablet 87459ddf48034b28b7b3e0d0d7de91f1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a82aad3b177f45fab1f9e9bcb6a5de57]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:51.217265 25239 tablet_service.cc:1467] Processing CreateTablet for tablet 87459ddf48034b28b7b3e0d0d7de91f1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a82aad3b177f45fab1f9e9bcb6a5de57]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:51.217761 25103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87459ddf48034b28b7b3e0d0d7de91f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:51.217777 25239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87459ddf48034b28b7b3e0d0d7de91f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:51.218047 25241 tablet_service.cc:1467] Processing CreateTablet for tablet 4127d80443f1499ab9a30935bbd4ca8e (DEFAULT_TABLE table=rebalance_test_table_9 [id=a82aad3b177f45fab1f9e9bcb6a5de57]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:51.218343 25241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4127d80443f1499ab9a30935bbd4ca8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:51.224659 25102 tablet_service.cc:1467] Processing CreateTablet for tablet 4127d80443f1499ab9a30935bbd4ca8e (DEFAULT_TABLE table=rebalance_test_table_9 [id=a82aad3b177f45fab1f9e9bcb6a5de57]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:51.224911 26655 tablet_bootstrap.cc:492] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c: Bootstrap starting.
I20240502 21:29:51.225046 25102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4127d80443f1499ab9a30935bbd4ca8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:51.225165 24971 tablet_service.cc:1467] Processing CreateTablet for tablet f801dcce3bfc42bf89a41156826a9203 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a82aad3b177f45fab1f9e9bcb6a5de57]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:51.225214 24969 tablet_service.cc:1467] Processing CreateTablet for tablet 87459ddf48034b28b7b3e0d0d7de91f1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a82aad3b177f45fab1f9e9bcb6a5de57]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:51.225603 24971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f801dcce3bfc42bf89a41156826a9203. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:51.225839 24969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87459ddf48034b28b7b3e0d0d7de91f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:51.226361 24837 tablet_service.cc:1467] Processing CreateTablet for tablet f801dcce3bfc42bf89a41156826a9203 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a82aad3b177f45fab1f9e9bcb6a5de57]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:51.226569 26655 tablet_bootstrap.cc:654] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:51.226837 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f801dcce3bfc42bf89a41156826a9203. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:51.228022 24970 tablet_service.cc:1467] Processing CreateTablet for tablet 4127d80443f1499ab9a30935bbd4ca8e (DEFAULT_TABLE table=rebalance_test_table_9 [id=a82aad3b177f45fab1f9e9bcb6a5de57]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:51.228367 24970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4127d80443f1499ab9a30935bbd4ca8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:51.228871 25104 tablet_service.cc:1467] Processing CreateTablet for tablet f801dcce3bfc42bf89a41156826a9203 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a82aad3b177f45fab1f9e9bcb6a5de57]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:51.229221 25104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f801dcce3bfc42bf89a41156826a9203. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:51.237264 26656 tablet_bootstrap.cc:492] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced: Bootstrap starting.
I20240502 21:29:51.239212 26656 tablet_bootstrap.cc:654] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:51.247493 26656 tablet_bootstrap.cc:492] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced: No bootstrap required, opened a new log
I20240502 21:29:51.247756 26656 ts_tablet_manager.cc:1367] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.010s
I20240502 21:29:51.248780 26656 raft_consensus.cc:348] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:51.249126 26656 raft_consensus.cc:374] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:51.249265 26656 raft_consensus.cc:729] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Initialized, Role: FOLLOWER
I20240502 21:29:51.249567 26656 consensus_queue.cc:260] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:51.250310 26656 ts_tablet_manager.cc:1398] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:29:51.250458 26659 tablet_bootstrap.cc:492] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557: Bootstrap starting.
I20240502 21:29:51.251338 26656 tablet_bootstrap.cc:492] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced: Bootstrap starting.
I20240502 21:29:51.252475 26660 tablet_bootstrap.cc:492] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b: Bootstrap starting.
I20240502 21:29:51.253496 26656 tablet_bootstrap.cc:654] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:51.256152 26660 tablet_bootstrap.cc:654] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:51.258623 26659 tablet_bootstrap.cc:654] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:51.258733 26655 tablet_bootstrap.cc:492] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c: No bootstrap required, opened a new log
I20240502 21:29:51.258934 26655 ts_tablet_manager.cc:1367] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c: Time spent bootstrapping tablet: real 0.035s user 0.013s sys 0.018s
I20240502 21:29:51.259502 26656 tablet_bootstrap.cc:492] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced: No bootstrap required, opened a new log
I20240502 21:29:51.259704 26656 ts_tablet_manager.cc:1367] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20240502 21:29:51.259938 26655 raft_consensus.cc:348] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:51.260224 26655 raft_consensus.cc:374] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:51.260322 26655 raft_consensus.cc:729] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Initialized, Role: FOLLOWER
I20240502 21:29:51.260762 26655 consensus_queue.cc:260] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:51.261593 26655 ts_tablet_manager.cc:1398] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:29:51.261898 26655 tablet_bootstrap.cc:492] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c: Bootstrap starting.
I20240502 21:29:51.262017 26656 raft_consensus.cc:348] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:51.262321 26656 raft_consensus.cc:374] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:51.262410 26656 raft_consensus.cc:729] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Initialized, Role: FOLLOWER
I20240502 21:29:51.262702 26656 consensus_queue.cc:260] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:51.263499 26656 ts_tablet_manager.cc:1398] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 21:29:51.264753 26660 tablet_bootstrap.cc:492] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b: No bootstrap required, opened a new log
I20240502 21:29:51.264957 26660 ts_tablet_manager.cc:1367] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20240502 21:29:51.265858 26656 tablet_bootstrap.cc:492] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced: Bootstrap starting.
I20240502 21:29:51.265810 26660 raft_consensus.cc:348] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:51.266155 26660 raft_consensus.cc:374] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:51.266304 26660 raft_consensus.cc:729] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Initialized, Role: FOLLOWER
I20240502 21:29:51.267519 26660 consensus_queue.cc:260] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:51.268826 26656 tablet_bootstrap.cc:654] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:51.269174 26660 ts_tablet_manager.cc:1398] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:29:51.271750 26655 tablet_bootstrap.cc:654] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:51.273406 26659 tablet_bootstrap.cc:492] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557: No bootstrap required, opened a new log
I20240502 21:29:51.273620 26659 ts_tablet_manager.cc:1367] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.013s
I20240502 21:29:51.274364 26656 tablet_bootstrap.cc:492] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced: No bootstrap required, opened a new log
I20240502 21:29:51.274565 26656 ts_tablet_manager.cc:1367] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20240502 21:29:51.274641 26659 raft_consensus.cc:348] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:51.274935 26659 raft_consensus.cc:374] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:51.275049 26659 raft_consensus.cc:729] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Initialized, Role: FOLLOWER
I20240502 21:29:51.275331 26659 consensus_queue.cc:260] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:51.275744 26656 raft_consensus.cc:348] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:51.276192 26656 raft_consensus.cc:374] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:51.276348 26656 raft_consensus.cc:729] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Initialized, Role: FOLLOWER
I20240502 21:29:51.276603 26659 ts_tablet_manager.cc:1398] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:29:51.276880 26659 tablet_bootstrap.cc:492] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557: Bootstrap starting.
I20240502 21:29:51.276655 26656 consensus_queue.cc:260] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:51.277216 26655 tablet_bootstrap.cc:492] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c: No bootstrap required, opened a new log
I20240502 21:29:51.277384 26655 ts_tablet_manager.cc:1367] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240502 21:29:51.278422 26656 ts_tablet_manager.cc:1398] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:29:51.278564 26659 tablet_bootstrap.cc:654] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:51.278501 26655 raft_consensus.cc:348] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:51.278740 26655 raft_consensus.cc:374] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:51.278816 26655 raft_consensus.cc:729] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Initialized, Role: FOLLOWER
I20240502 21:29:51.279052 26655 consensus_queue.cc:260] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:51.279649 26655 ts_tablet_manager.cc:1398] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:29:51.283947 26659 tablet_bootstrap.cc:492] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557: No bootstrap required, opened a new log
I20240502 21:29:51.284135 26659 ts_tablet_manager.cc:1367] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.002s
I20240502 21:29:51.284951 26659 raft_consensus.cc:348] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:51.285372 26659 raft_consensus.cc:374] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:51.285470 26659 raft_consensus.cc:729] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Initialized, Role: FOLLOWER
I20240502 21:29:51.285748 26659 consensus_queue.cc:260] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:51.286196 26659 ts_tablet_manager.cc:1398] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:29:51.286564 26659 tablet_bootstrap.cc:492] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557: Bootstrap starting.
I20240502 21:29:51.288149 26659 tablet_bootstrap.cc:654] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:51.288499 26642 raft_consensus.cc:482] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:51.288657 26642 raft_consensus.cc:504] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:51.289523 26642 leader_election.cc:288] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:51.289963 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4127d80443f1499ab9a30935bbd4ca8e" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557" is_pre_election: true
W20240502 21:29:51.290645 25058 leader_election.cc:341] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:29:51.290957 26659 tablet_bootstrap.cc:492] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557: No bootstrap required, opened a new log
I20240502 21:29:51.291252 26659 ts_tablet_manager.cc:1367] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 21:29:51.291191 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4127d80443f1499ab9a30935bbd4ca8e" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b7d5b9696b4046b522c880c748fe6c" is_pre_election: true
I20240502 21:29:51.291417 25260 raft_consensus.cc:2454] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 0.
I20240502 21:29:51.291762 25059 leader_election.cc:302] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced; no voters: 64d97dd1674d43ccadd7b4a38eeff557
I20240502 21:29:51.292018 26642 raft_consensus.cc:2781] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:51.292022 26659 raft_consensus.cc:348] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:51.292145 26642 raft_consensus.cc:482] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:51.292255 26642 raft_consensus.cc:3037] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:51.292253 26659 raft_consensus.cc:374] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:51.292348 26659 raft_consensus.cc:729] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Initialized, Role: FOLLOWER
I20240502 21:29:51.292572 26659 consensus_queue.cc:260] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:51.293123 26659 ts_tablet_manager.cc:1398] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:29:51.294437 26642 raft_consensus.cc:504] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:51.294996 26642 leader_election.cc:288] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:51.295468 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4127d80443f1499ab9a30935bbd4ca8e" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:51.295469 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4127d80443f1499ab9a30935bbd4ca8e" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:51.295728 25260 raft_consensus.cc:3037] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:51.295743 24992 raft_consensus.cc:3037] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:51.298549 25260 raft_consensus.cc:2454] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 1.
I20240502 21:29:51.298548 24992 raft_consensus.cc:2454] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 1.
I20240502 21:29:51.298942 25058 leader_election.cc:302] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:51.299253 26642 raft_consensus.cc:2781] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:51.299419 26642 raft_consensus.cc:686] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 1 LEADER]: Becoming Leader. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:51.299746 26642 consensus_queue.cc:237] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:51.302879 24703 catalog_manager.cc:5484] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bf90e514781418da921c046aa5d9ced (127.29.82.67). New cstate: current_term: 1 leader_uuid: "5bf90e514781418da921c046aa5d9ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:51.330816 26535 raft_consensus.cc:482] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:51.331000 26535 raft_consensus.cc:504] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:51.331519 26535 leader_election.cc:288] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:51.332072 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f801dcce3bfc42bf89a41156826a9203" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b" is_pre_election: true
I20240502 21:29:51.332103 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f801dcce3bfc42bf89a41156826a9203" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced" is_pre_election: true
I20240502 21:29:51.332324 24856 raft_consensus.cc:2454] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 0.
I20240502 21:29:51.332340 25123 raft_consensus.cc:2454] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 0.
I20240502 21:29:51.332794 24926 leader_election.cc:302] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:51.333097 26535 raft_consensus.cc:2781] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:51.333191 26535 raft_consensus.cc:482] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:51.333317 26535 raft_consensus.cc:3037] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:51.335194 26535 raft_consensus.cc:504] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:51.335770 26535 leader_election.cc:288] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 1 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:51.336216 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f801dcce3bfc42bf89a41156826a9203" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:51.336439 24856 raft_consensus.cc:3037] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:51.336405 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f801dcce3bfc42bf89a41156826a9203" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:51.336616 25123 raft_consensus.cc:3037] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:51.337200 26535 raft_consensus.cc:482] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:51.337363 26535 raft_consensus.cc:504] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:51.337990 26535 leader_election.cc:288] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:51.338444 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87459ddf48034b28b7b3e0d0d7de91f1" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b7d5b9696b4046b522c880c748fe6c" is_pre_election: true
I20240502 21:29:51.338691 25260 raft_consensus.cc:2454] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 0.
I20240502 21:29:51.338920 25123 raft_consensus.cc:2454] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 1.
I20240502 21:29:51.339098 24856 raft_consensus.cc:2454] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 1.
I20240502 21:29:51.339145 24927 leader_election.cc:302] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:51.339372 24926 leader_election.cc:302] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:51.339447 26535 raft_consensus.cc:2781] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:51.339540 26535 raft_consensus.cc:482] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:51.339740 26535 raft_consensus.cc:3037] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:51.339768 26644 raft_consensus.cc:2781] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:51.340071 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87459ddf48034b28b7b3e0d0d7de91f1" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced" is_pre_election: true
I20240502 21:29:51.340337 25123 raft_consensus.cc:2454] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 0.
I20240502 21:29:51.340420 26644 raft_consensus.cc:686] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:51.340826 26644 consensus_queue.cc:237] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:51.342200 26535 raft_consensus.cc:504] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:51.343235 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87459ddf48034b28b7b3e0d0d7de91f1" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:51.343416 26535 leader_election.cc:288] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 1 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:51.343446 25260 raft_consensus.cc:3037] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:51.343497 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87459ddf48034b28b7b3e0d0d7de91f1" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:51.343719 25123 raft_consensus.cc:3037] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:51.344647 24703 catalog_manager.cc:5484] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:51.345821 25123 raft_consensus.cc:2454] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 1.
I20240502 21:29:51.346134 25260 raft_consensus.cc:2454] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 1.
I20240502 21:29:51.346483 24926 leader_election.cc:302] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:51.347034 26535 raft_consensus.cc:2781] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:51.347338 26535 raft_consensus.cc:686] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:51.347635 26535 consensus_queue.cc:237] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:51.350597 24703 catalog_manager.cc:5484] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 1 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:51.714236 26642 consensus_queue.cc:1035] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:29:51.715678 26672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:51.716074 26672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 21:29:51.721050 26642 consensus_queue.cc:1035] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:29:51.728518 26672 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240502 21:29:51.816854 26535 consensus_queue.cc:1035] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:51.822810 26669 consensus_queue.cc:1035] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:51.881798 24856 raft_consensus.cc:1261] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Refusing update from remote peer 64d97dd1674d43ccadd7b4a38eeff557: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:29:51.882730 26669 consensus_queue.cc:1035] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:51.884861 25123 raft_consensus.cc:1261] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Refusing update from remote peer 64d97dd1674d43ccadd7b4a38eeff557: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:29:51.885993 26669 consensus_queue.cc:1035] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:51.973897 26718 mvcc.cc:205] Tried to move back new op lower bound from 7023351365105545216 to 7023351362934845440. Current Snapshot: MvccSnapshot[applied={T|T < 7023351365105545216}]
INSERT report
rows total: 2000
time total: 228.889 ms
time per row: 0.114444 ms
I20240502 21:29:52.125068 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.69:46087
--local_ip_for_outbound_sockets=127.29.82.69
--tserver_master_addrs=127.29.82.126:45283
--webserver_port=38963
--webserver_interface=127.29.82.69
--builtin_ntp_servers=127.29.82.84:41041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 21:29:52.469151 26733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:52.469473 26733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:29:52.469614 26733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:52.477406 26733 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:29:52.477553 26733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:29:52.477723 26733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.69
I20240502 21:29:52.488379 26733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:41041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.69:46087
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.29.82.69
--webserver_port=38963
--tserver_master_addrs=127.29.82.126:45283
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.69
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:29:52.490013 26733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:29:52.492318 26733 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:29:52.506175 26739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:52.506709 26740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:52.507424 26742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:29:52.508000 26733 server_base.cc:1034] running on GCE node
I20240502 21:29:52.508764 26733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:29:52.509847 26733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:29:52.511080 26733 hybrid_clock.cc:648] HybridClock initialized: now 1714685392511035 us; error 39 us; skew 500 ppm
I20240502 21:29:52.511461 26733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:29:52.518340 26733 webserver.cc:460] Webserver started at http://127.29.82.69:38963/ using document root <none> and password file <none>
I20240502 21:29:52.519079 26733 fs_manager.cc:359] Metadata directory not provided
I20240502 21:29:52.519219 26733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:29:52.525003 26733 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20240502 21:29:52.528193 26748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:52.529547 26733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:29:52.529708 26733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "0b56aba0e3e9491c9f5e9ee2b281e316"
format_stamp: "Formatted at 2024-05-02 21:29:34 on dist-test-slave-xzsm"
I20240502 21:29:52.530508 26733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:29:52.542695 26733 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:29:52.543391 26733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:29:52.543694 26733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:29:52.544584 26733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:29:52.546316 26733 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:29:52.546415 26733 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:52.546550 26733 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:29:52.546607 26733 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:52.587780 26733 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.69:46087
I20240502 21:29:52.587862 26864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.69:46087 every 8 connection(s)
I20240502 21:29:52.589691 26733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:29:52.598829 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 26733
I20240502 21:29:52.599623 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.70:36971
--local_ip_for_outbound_sockets=127.29.82.70
--tserver_master_addrs=127.29.82.126:45283
--webserver_port=43687
--webserver_interface=127.29.82.70
--builtin_ntp_servers=127.29.82.84:41041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:29:52.604422 26865 heartbeater.cc:347] Connected to a master server at 127.29.82.126:45283
I20240502 21:29:52.604732 26865 heartbeater.cc:464] Registering TS with master...
I20240502 21:29:52.605532 26865 heartbeater.cc:511] Master 127.29.82.126:45283 requested a full tablet report, sending...
I20240502 21:29:52.607120 24703 ts_manager.cc:194] Re-registered known tserver with Master: 0b56aba0e3e9491c9f5e9ee2b281e316 (127.29.82.69:46087)
I20240502 21:29:52.608122 24703 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.69:37667
W20240502 21:29:52.952432 26869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:52.952718 26869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:29:52.952857 26869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:52.960239 26869 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:29:52.960397 26869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:29:52.960556 26869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.70
I20240502 21:29:52.970427 26869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:41041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.70:36971
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.29.82.70
--webserver_port=43687
--tserver_master_addrs=127.29.82.126:45283
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.70
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:29:52.971798 26869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:29:52.973567 26869 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:29:52.987175 26877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:52.987615 26880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:52.988301 26878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:29:52.988970 26869 server_base.cc:1034] running on GCE node
I20240502 21:29:52.989651 26869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:29:52.991403 26869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:29:52.992663 26869 hybrid_clock.cc:648] HybridClock initialized: now 1714685392992553 us; error 94 us; skew 500 ppm
I20240502 21:29:52.993103 26869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:29:52.995736 26869 webserver.cc:460] Webserver started at http://127.29.82.70:43687/ using document root <none> and password file <none>
I20240502 21:29:52.996479 26869 fs_manager.cc:359] Metadata directory not provided
I20240502 21:29:52.996614 26869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:29:53.002286 26869 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20240502 21:29:53.005316 26886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:53.006711 26869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 21:29:53.006893 26869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "506baf9b2c0a44c28c3e7fd08733f795"
format_stamp: "Formatted at 2024-05-02 21:29:34 on dist-test-slave-xzsm"
I20240502 21:29:53.007793 26869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:29:53.043330 26869 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:29:53.044217 26869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:29:53.044494 26869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:29:53.045485 26869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:29:53.047009 26869 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:29:53.047084 26869 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:53.047183 26869 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:29:53.047226 26869 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:53.081673 26869 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.70:36971
I20240502 21:29:53.081722 27000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.70:36971 every 8 connection(s)
I20240502 21:29:53.083179 26869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 21:29:53.085631 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 26869
I20240502 21:29:53.086313 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.71:36399
--local_ip_for_outbound_sockets=127.29.82.71
--tserver_master_addrs=127.29.82.126:45283
--webserver_port=40287
--webserver_interface=127.29.82.71
--builtin_ntp_servers=127.29.82.84:41041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:29:53.097226 27001 heartbeater.cc:347] Connected to a master server at 127.29.82.126:45283
I20240502 21:29:53.097568 27001 heartbeater.cc:464] Registering TS with master...
I20240502 21:29:53.098493 27001 heartbeater.cc:511] Master 127.29.82.126:45283 requested a full tablet report, sending...
I20240502 21:29:53.100140 24703 ts_manager.cc:194] Re-registered known tserver with Master: 506baf9b2c0a44c28c3e7fd08733f795 (127.29.82.70:36971)
I20240502 21:29:53.101027 24703 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.70:39553
W20240502 21:29:53.447670 27005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:53.447973 27005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:29:53.448107 27005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:53.455936 27005 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:29:53.456053 27005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:29:53.456216 27005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.71
I20240502 21:29:53.465974 27005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:41041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.71:36399
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.29.82.71
--webserver_port=40287
--tserver_master_addrs=127.29.82.126:45283
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.71
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:29:53.467422 27005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:29:53.469209 27005 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:29:53.482856 27014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:53.483184 27012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:53.483866 27011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:29:53.484645 27005 server_base.cc:1034] running on GCE node
I20240502 21:29:53.485384 27005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:29:53.486582 27005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:29:53.487804 27005 hybrid_clock.cc:648] HybridClock initialized: now 1714685393487743 us; error 44 us; skew 500 ppm
I20240502 21:29:53.488190 27005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:29:53.490813 27005 webserver.cc:460] Webserver started at http://127.29.82.71:40287/ using document root <none> and password file <none>
I20240502 21:29:53.491729 27005 fs_manager.cc:359] Metadata directory not provided
I20240502 21:29:53.491876 27005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:29:53.498706 27005 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 21:29:53.502111 27020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:53.503674 27005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:29:53.503844 27005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "332de10467624bdb8cdcb69669543330"
format_stamp: "Formatted at 2024-05-02 21:29:35 on dist-test-slave-xzsm"
I20240502 21:29:53.504719 27005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:29:53.524901 27005 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:29:53.525712 27005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:29:53.525975 27005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:29:53.527012 27005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:29:53.528555 27005 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:29:53.528630 27005 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:53.528729 27005 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:29:53.528774 27005 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:53.564016 27005 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.71:36399
I20240502 21:29:53.564095 27133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.71:36399 every 8 connection(s)
I20240502 21:29:53.565497 27005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 21:29:53.574261 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 27005
I20240502 21:29:53.574856 30025 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
/tmp/dist-test-taskbqdCfi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.82.72:44277
--local_ip_for_outbound_sockets=127.29.82.72
--tserver_master_addrs=127.29.82.126:45283
--webserver_port=35125
--webserver_interface=127.29.82.72
--builtin_ntp_servers=127.29.82.84:41041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:29:53.581549 27134 heartbeater.cc:347] Connected to a master server at 127.29.82.126:45283
I20240502 21:29:53.581923 27134 heartbeater.cc:464] Registering TS with master...
I20240502 21:29:53.582947 27134 heartbeater.cc:511] Master 127.29.82.126:45283 requested a full tablet report, sending...
I20240502 21:29:53.584661 24703 ts_manager.cc:194] Re-registered known tserver with Master: 332de10467624bdb8cdcb69669543330 (127.29.82.71:36399)
I20240502 21:29:53.585556 24703 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.71:53389
I20240502 21:29:53.611172 26865 heartbeater.cc:503] Master 127.29.82.126:45283 was elected leader, sending a full tablet report...
W20240502 21:29:53.940721 27137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:53.941027 27137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:29:53.941169 27137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:53.948639 27137 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:29:53.948765 27137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:29:53.948884 27137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.82.72
I20240502 21:29:53.958715 27137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.82.84:41041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.82.72:44277
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.29.82.72
--webserver_port=35125
--tserver_master_addrs=127.29.82.126:45283
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.82.72
--log_dir=/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:29:53.960062 27137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:29:53.961834 27137 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:29:53.975881 27143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:53.976483 27142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:53.977099 27145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:29:53.977864 27137 server_base.cc:1034] running on GCE node
I20240502 21:29:53.978523 27137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:29:53.979544 27137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:29:53.980808 27137 hybrid_clock.cc:648] HybridClock initialized: now 1714685393980749 us; error 48 us; skew 500 ppm
I20240502 21:29:53.981168 27137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:29:53.983440 27137 webserver.cc:460] Webserver started at http://127.29.82.72:35125/ using document root <none> and password file <none>
I20240502 21:29:53.984347 27137 fs_manager.cc:359] Metadata directory not provided
I20240502 21:29:53.984494 27137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:29:53.990500 27137 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 21:29:53.993798 27152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:53.995362 27137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20240502 21:29:53.995553 27137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "eb2ceee828af41b8bb7f9bf11fc2c573"
format_stamp: "Formatted at 2024-05-02 21:29:35 on dist-test-slave-xzsm"
I20240502 21:29:53.996569 27137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:29:54.014487 27137 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:29:54.015358 27137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:29:54.015738 27137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:29:54.016978 27137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:29:54.018644 27137 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:29:54.018729 27137 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:54.018872 27137 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:29:54.018931 27137 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:54.072116 27137 rpc_server.cc:307] RPC server started. Bound to: 127.29.82.72:44277
I20240502 21:29:54.072156 27267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.82.72:44277 every 8 connection(s)
I20240502 21:29:54.073710 27137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbqdCfi/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714685124710458-30025-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 21:29:54.077757 30025 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbqdCfi/build/asan/bin/kudu as pid 27137
I20240502 21:29:54.090453 27268 heartbeater.cc:347] Connected to a master server at 127.29.82.126:45283
I20240502 21:29:54.090783 27268 heartbeater.cc:464] Registering TS with master...
I20240502 21:29:54.091825 27268 heartbeater.cc:511] Master 127.29.82.126:45283 requested a full tablet report, sending...
I20240502 21:29:54.093607 24703 ts_manager.cc:194] Re-registered known tserver with Master: eb2ceee828af41b8bb7f9bf11fc2c573 (127.29.82.72:44277)
I20240502 21:29:54.095209 24703 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.82.72:42523
I20240502 21:29:54.103976 27001 heartbeater.cc:503] Master 127.29.82.126:45283 was elected leader, sending a full tablet report...
I20240502 21:29:54.125041 24991 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4127d80443f1499ab9a30935bbd4ca8e"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:54.125308 24991 raft_consensus.cc:482] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.125425 24991 raft_consensus.cc:3037] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.127416 24991 raft_consensus.cc:504] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.128088 24991 leader_election.cc:288] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:54.128643 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4127d80443f1499ab9a30935bbd4ca8e" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.128705 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4127d80443f1499ab9a30935bbd4ca8e" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:54.128885 25123 raft_consensus.cc:3032] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:54.128959 25260 raft_consensus.cc:3037] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.129006 25123 raft_consensus.cc:729] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:54.129323 25123 consensus_queue.cc:260] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.129441 24991 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f801dcce3bfc42bf89a41156826a9203"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:54.129626 24991 raft_consensus.cc:465] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 21:29:54.129886 25123 raft_consensus.cc:3037] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.131312 25260 raft_consensus.cc:2454] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.131753 24927 leader_election.cc:302] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:54.132043 27151 raft_consensus.cc:2781] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:54.132158 27151 raft_consensus.cc:686] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.132438 27151 consensus_queue.cc:237] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.132519 25123 raft_consensus.cc:2454] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.133038 24992 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "87459ddf48034b28b7b3e0d0d7de91f1"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:54.133234 24992 raft_consensus.cc:465] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 21:29:54.134277 24992 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bdecfa03390e405ca4925f8d1a40e0ca"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:54.134474 24992 raft_consensus.cc:482] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.134621 24992 raft_consensus.cc:3037] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.136359 24703 catalog_manager.cc:5484] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 1 to 2, leader changed from 5bf90e514781418da921c046aa5d9ced (127.29.82.67) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 2 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:54.137327 24992 raft_consensus.cc:504] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.138132 24992 leader_election.cc:288] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:54.139113 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdecfa03390e405ca4925f8d1a40e0ca" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:54.139386 24857 raft_consensus.cc:3037] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.142308 24857 raft_consensus.cc:2454] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.142957 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdecfa03390e405ca4925f8d1a40e0ca" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:54.143186 25260 raft_consensus.cc:3032] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:54.143296 25260 raft_consensus.cc:729] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:54.143640 24926 leader_election.cc:302] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:54.143643 25260 consensus_queue.cc:260] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.144058 27151 raft_consensus.cc:2781] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:54.144204 25260 raft_consensus.cc:3037] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.144253 27151 raft_consensus.cc:686] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.144564 27151 consensus_queue.cc:237] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.147115 24992 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3c1ce7cf99164a9a848cbb22ad2052ea"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:54.146826 25260 raft_consensus.cc:2454] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.148077 24992 raft_consensus.cc:482] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.148300 24992 raft_consensus.cc:3037] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.152309 24992 raft_consensus.cc:504] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.153738 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1ce7cf99164a9a848cbb22ad2052ea" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:54.153966 24857 raft_consensus.cc:3037] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.154387 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1ce7cf99164a9a848cbb22ad2052ea" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.154625 25123 raft_consensus.cc:3032] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:54.154747 25123 raft_consensus.cc:729] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:54.155071 25123 consensus_queue.cc:260] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.155655 25123 raft_consensus.cc:3037] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.156777 24857 raft_consensus.cc:2454] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.157867 24926 leader_election.cc:302] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:54.158120 24992 leader_election.cc:288] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:54.158449 27151 raft_consensus.cc:2781] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:54.158627 27151 raft_consensus.cc:686] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.158916 27151 consensus_queue.cc:237] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.159417 24991 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f62981fdfb2349bd9fbe0e85b77ae492"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:54.159605 24991 raft_consensus.cc:482] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.159731 24991 raft_consensus.cc:3037] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.163803 25123 raft_consensus.cc:2454] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.163396 24991 raft_consensus.cc:504] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.165334 24991 leader_election.cc:288] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:54.166136 25122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62981fdfb2349bd9fbe0e85b77ae492" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.166375 25122 raft_consensus.cc:3032] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:54.166489 25122 raft_consensus.cc:729] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:54.166802 25122 consensus_queue.cc:260] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.167281 25122 raft_consensus.cc:3037] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.168628 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62981fdfb2349bd9fbe0e85b77ae492" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:54.168918 24857 raft_consensus.cc:3037] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.170665 25122 raft_consensus.cc:2454] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.173713 24857 raft_consensus.cc:2454] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.174209 24926 leader_election.cc:302] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:54.174551 26535 raft_consensus.cc:2781] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:54.174975 24991 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d643c15afbec4933a0d0a145f17f703c"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:54.175205 24991 raft_consensus.cc:482] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.175318 24991 raft_consensus.cc:3037] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.178383 24991 raft_consensus.cc:504] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.179494 24991 leader_election.cc:288] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:54.179893 25122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d643c15afbec4933a0d0a145f17f703c" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.180155 25122 raft_consensus.cc:3032] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:54.180271 25122 raft_consensus.cc:729] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:54.180529 24992 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57b40e72e8324b4ba6febf5eda24d333"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:54.180704 24992 raft_consensus.cc:465] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 21:29:54.180600 25122 consensus_queue.cc:260] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.181082 25122 raft_consensus.cc:3037] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.181654 24992 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "69afb121cb564de3a9bec5e3f1cc0492"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:54.181828 24992 raft_consensus.cc:482] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.181952 24992 raft_consensus.cc:3037] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.184464 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d643c15afbec4933a0d0a145f17f703c" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:54.184736 25260 raft_consensus.cc:3037] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.184772 24992 raft_consensus.cc:504] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.185559 24992 leader_election.cc:288] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:54.186231 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69afb121cb564de3a9bec5e3f1cc0492" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.186412 25123 raft_consensus.cc:3037] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.188654 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69afb121cb564de3a9bec5e3f1cc0492" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:54.188926 25261 raft_consensus.cc:3032] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:54.189019 25261 raft_consensus.cc:729] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:54.189010 25123 raft_consensus.cc:2454] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.189404 25261 consensus_queue.cc:260] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.189893 25261 raft_consensus.cc:3037] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.190088 24992 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a3a378f7a44648c3accc9dcba847040d"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:54.190274 24992 raft_consensus.cc:482] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.190372 24992 raft_consensus.cc:3037] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.193179 24992 raft_consensus.cc:504] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.193900 24992 leader_election.cc:288] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:54.194335 25261 raft_consensus.cc:2454] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.194806 24927 leader_election.cc:302] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:54.195621 27287 raft_consensus.cc:2781] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:54.195799 25260 raft_consensus.cc:2454] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.195834 27287 raft_consensus.cc:686] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.196138 27287 consensus_queue.cc:237] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.196467 25122 raft_consensus.cc:2454] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.197435 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a378f7a44648c3accc9dcba847040d" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:54.197734 24857 raft_consensus.cc:3032] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:54.197867 24857 raft_consensus.cc:729] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.197804 24703 catalog_manager.cc:5484] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 1 to 2, leader changed from 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 2 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } }
I20240502 21:29:54.198210 24991 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "89c4eee02b444dc188172fd8c7b411bb"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:54.198357 24991 raft_consensus.cc:482] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.198269 24857 consensus_queue.cc:260] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.198472 24991 raft_consensus.cc:3037] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.198379 24703 catalog_manager.cc:5484] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 1 to 2, leader changed from 5bf90e514781418da921c046aa5d9ced (127.29.82.67) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 2 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:54.198951 24857 raft_consensus.cc:3037] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.201015 24991 raft_consensus.cc:504] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.201836 24927 leader_election.cc:302] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:54.201948 24991 leader_election.cc:288] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:54.202086 24857 raft_consensus.cc:2454] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.202543 27151 raft_consensus.cc:2781] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:54.202591 25122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a378f7a44648c3accc9dcba847040d" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.202693 27151 raft_consensus.cc:686] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.202786 25122 raft_consensus.cc:3037] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.202955 27151 consensus_queue.cc:237] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.206274 25122 raft_consensus.cc:2454] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.207684 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c4eee02b444dc188172fd8c7b411bb" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:54.207981 25261 raft_consensus.cc:3032] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:54.208089 25261 raft_consensus.cc:729] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:54.208446 25261 consensus_queue.cc:260] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.208843 25261 raft_consensus.cc:3037] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.212268 25122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c4eee02b444dc188172fd8c7b411bb" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.212527 25122 raft_consensus.cc:3037] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.214038 24926 leader_election.cc:302] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:54.214305 27287 raft_consensus.cc:2781] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:54.214459 27287 raft_consensus.cc:686] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.214746 27287 consensus_queue.cc:237] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.215229 25122 raft_consensus.cc:2454] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.217362 24926 leader_election.cc:302] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:54.217851 27288 raft_consensus.cc:2781] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:54.217924 24991 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e3d3eab014a466fa4a421f7d9d20fba"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:54.217993 27288 raft_consensus.cc:686] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.218119 24991 raft_consensus.cc:482] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.218225 24991 raft_consensus.cc:3037] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.218328 27288 consensus_queue.cc:237] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.221908 24991 raft_consensus.cc:504] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.222712 24991 leader_election.cc:288] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:54.224197 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3d3eab014a466fa4a421f7d9d20fba" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.224436 25123 raft_consensus.cc:3037] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.226228 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3d3eab014a466fa4a421f7d9d20fba" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:54.226430 24857 raft_consensus.cc:3032] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:54.226518 24857 raft_consensus.cc:729] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.226783 24857 consensus_queue.cc:260] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.227056 25123 raft_consensus.cc:2454] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.229319 24857 raft_consensus.cc:3037] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.233484 24857 raft_consensus.cc:2454] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.234730 25261 raft_consensus.cc:2454] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.235805 24992 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "175511733b714d2ea3cf877a0e5df418"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:54.236032 24992 raft_consensus.cc:465] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 21:29:54.236465 24926 leader_election.cc:302] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:54.237357 27288 raft_consensus.cc:2781] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:54.237586 27288 raft_consensus.cc:686] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.237757 24991 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a851d0bede1444c8a96ea45ea948bd59"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:54.238094 24991 raft_consensus.cc:482] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.238240 24991 raft_consensus.cc:3037] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.238008 27288 consensus_queue.cc:237] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.245661 24991 raft_consensus.cc:504] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.247350 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851d0bede1444c8a96ea45ea948bd59" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:54.247654 24857 raft_consensus.cc:3037] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.247965 26535 raft_consensus.cc:686] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.248317 26535 consensus_queue.cc:237] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.250170 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851d0bede1444c8a96ea45ea948bd59" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:54.250442 24857 raft_consensus.cc:2454] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.250480 25261 raft_consensus.cc:3032] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:54.250610 25261 raft_consensus.cc:729] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:54.250923 24926 leader_election.cc:302] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:54.251160 25261 consensus_queue.cc:260] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.251487 25261 raft_consensus.cc:3037] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.253564 25261 raft_consensus.cc:2454] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.254024 27286 raft_consensus.cc:2781] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:54.254199 27286 raft_consensus.cc:686] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.254500 27286 consensus_queue.cc:237] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.255088 24991 leader_election.cc:288] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:54.256335 24991 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3e32a87a85274194916bef376c7e558e"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:54.256623 24991 raft_consensus.cc:482] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.256820 24991 raft_consensus.cc:3037] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.259109 24703 catalog_manager.cc:5484] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 1 to 2, leader changed from 5bf90e514781418da921c046aa5d9ced (127.29.82.67) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 2 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } }
I20240502 21:29:54.259706 24703 catalog_manager.cc:5484] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 1 to 2, leader changed from 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 2 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:54.260174 24703 catalog_manager.cc:5484] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 1 to 2, leader changed from 5bf90e514781418da921c046aa5d9ced (127.29.82.67) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 2 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:54.260643 24991 raft_consensus.cc:504] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.261822 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e32a87a85274194916bef376c7e558e" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.262039 25123 raft_consensus.cc:3032] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:54.262144 25123 raft_consensus.cc:729] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:54.262413 25123 consensus_queue.cc:260] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.262800 25123 raft_consensus.cc:3037] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.267294 25123 raft_consensus.cc:2454] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.267757 24926 leader_election.cc:302] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:54.268060 27151 raft_consensus.cc:2781] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:54.268227 27151 raft_consensus.cc:686] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.268507 27151 consensus_queue.cc:237] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.268888 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e32a87a85274194916bef376c7e558e" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:54.269264 24857 raft_consensus.cc:3037] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.273120 24991 leader_election.cc:288] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:54.274631 24992 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c574664a57444c95b70d754cc46e313e"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:54.274832 24992 raft_consensus.cc:465] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 21:29:54.274821 24857 raft_consensus.cc:2454] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.276309 24992 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f25dfc6b0c04d878cd3db58d37cddec"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:54.276459 24992 raft_consensus.cc:482] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.276585 24992 raft_consensus.cc:3037] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.276352 24703 catalog_manager.cc:5484] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 1 to 2, leader changed from 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 2 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:54.276851 24703 catalog_manager.cc:5484] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 1 to 2, leader changed from 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 2 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:54.277288 24703 catalog_manager.cc:5484] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 1 to 2, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 2 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } }
I20240502 21:29:54.277685 24703 catalog_manager.cc:5484] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 1 to 2, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 2 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:54.278777 24992 raft_consensus.cc:504] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.279831 24992 leader_election.cc:288] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:54.279888 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f25dfc6b0c04d878cd3db58d37cddec" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:54.280121 25261 raft_consensus.cc:3037] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.280202 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f25dfc6b0c04d878cd3db58d37cddec" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.280395 25123 raft_consensus.cc:3032] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:54.280494 25123 raft_consensus.cc:729] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:54.280788 25123 consensus_queue.cc:260] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.281178 25123 raft_consensus.cc:3037] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.283039 25123 raft_consensus.cc:2454] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.283139 25261 raft_consensus.cc:2454] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.283453 24926 leader_election.cc:302] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:54.283762 27151 raft_consensus.cc:2781] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:54.283946 27151 raft_consensus.cc:686] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.284210 27151 consensus_queue.cc:237] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.284758 24992 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "69df123d92d74693b4654c6a809bbe89"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:54.284991 24992 raft_consensus.cc:482] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.285127 24992 raft_consensus.cc:3037] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.287941 24992 raft_consensus.cc:504] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.288501 24703 catalog_manager.cc:5484] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 1 to 2, leader changed from 5bf90e514781418da921c046aa5d9ced (127.29.82.67) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 2 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:54.288936 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69df123d92d74693b4654c6a809bbe89" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.288978 24703 catalog_manager.cc:5484] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 1 to 2, leader changed from 5bf90e514781418da921c046aa5d9ced (127.29.82.67) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 2 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:54.289213 25123 raft_consensus.cc:3032] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:54.289059 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69df123d92d74693b4654c6a809bbe89" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:54.289326 25123 raft_consensus.cc:729] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:54.289595 25261 raft_consensus.cc:3037] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.289600 25123 consensus_queue.cc:260] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.290024 25123 raft_consensus.cc:3037] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.292843 25261 raft_consensus.cc:2454] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.292976 25123 raft_consensus.cc:2454] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.293416 24927 leader_election.cc:302] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:54.293582 24992 leader_election.cc:288] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:54.294152 27151 raft_consensus.cc:2781] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:54.294363 27151 raft_consensus.cc:686] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.294832 24992 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fed2d260d00d470e95e9a62cc71ba94f"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:54.294905 27151 consensus_queue.cc:237] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.295035 24992 raft_consensus.cc:482] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.295171 24992 raft_consensus.cc:3037] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.298111 24703 catalog_manager.cc:5484] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 1 to 2, leader changed from 5bf90e514781418da921c046aa5d9ced (127.29.82.67) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 2 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:54.298336 24992 raft_consensus.cc:504] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.299604 24992 leader_election.cc:288] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:54.299468 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fed2d260d00d470e95e9a62cc71ba94f" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:54.299836 24857 raft_consensus.cc:3037] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.300668 24992 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "52153a02d80043f0a1840b0e5aadd9b3"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:54.300981 24992 raft_consensus.cc:465] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 21:29:54.302093 24992 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "986c305824414299ac30f1e50f514c8a"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:54.302374 24992 raft_consensus.cc:482] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.302577 24992 raft_consensus.cc:3037] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.302575 24857 raft_consensus.cc:2454] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.303114 24926 leader_election.cc:302] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:54.303496 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fed2d260d00d470e95e9a62cc71ba94f" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:54.303738 25261 raft_consensus.cc:3032] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:54.303817 25261 raft_consensus.cc:729] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:54.304239 27151 raft_consensus.cc:2781] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:54.304600 25261 consensus_queue.cc:260] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.305019 25261 raft_consensus.cc:3037] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.305866 24992 raft_consensus.cc:504] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.306550 24992 leader_election.cc:288] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:54.306878 27151 raft_consensus.cc:686] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.306957 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c305824414299ac30f1e50f514c8a" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:54.307163 25260 raft_consensus.cc:3037] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.307157 27151 consensus_queue.cc:237] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.309523 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c305824414299ac30f1e50f514c8a" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:54.309688 25261 raft_consensus.cc:2454] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.309825 25260 raft_consensus.cc:2454] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.309854 24857 raft_consensus.cc:3032] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:54.310031 24857 raft_consensus.cc:729] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.310458 24927 leader_election.cc:302] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:54.310417 24857 consensus_queue.cc:260] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.310786 27286 raft_consensus.cc:2781] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:54.310945 27286 raft_consensus.cc:686] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.310950 24857 raft_consensus.cc:3037] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.311177 27286 consensus_queue.cc:237] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.314564 24857 raft_consensus.cc:2454] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.315063 24992 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8423ff21d93741cdab3093ea454bb532"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:54.315297 24992 raft_consensus.cc:482] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.315466 24992 raft_consensus.cc:3037] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.316241 24703 catalog_manager.cc:5484] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 1 to 2, leader changed from 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 2 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:54.316843 24703 catalog_manager.cc:5484] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 1 to 2, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 2 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:54.318356 24992 raft_consensus.cc:504] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.320606 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8423ff21d93741cdab3093ea454bb532" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:54.320849 25261 raft_consensus.cc:3037] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.323566 25261 raft_consensus.cc:2454] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.323812 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8423ff21d93741cdab3093ea454bb532" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:54.324051 24857 raft_consensus.cc:3032] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:54.323999 24927 leader_election.cc:302] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:54.324141 24857 raft_consensus.cc:729] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.324479 27286 raft_consensus.cc:2781] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:54.324429 24857 consensus_queue.cc:260] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.324626 27286 raft_consensus.cc:686] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.324802 24857 raft_consensus.cc:3037] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.324901 27286 consensus_queue.cc:237] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.323812 24992 leader_election.cc:288] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:54.327236 24992 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "48e7c6534b544336ae216d3200f4f01f"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:54.327437 24992 raft_consensus.cc:482] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.327554 24992 raft_consensus.cc:3037] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.328982 24703 catalog_manager.cc:5484] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 1 to 2, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 2 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:54.329274 24857 raft_consensus.cc:2454] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.330253 24992 raft_consensus.cc:504] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.330909 24992 leader_election.cc:288] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:54.331322 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e7c6534b544336ae216d3200f4f01f" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:54.331548 25261 raft_consensus.cc:3032] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:54.331692 25261 raft_consensus.cc:729] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:54.331743 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e7c6534b544336ae216d3200f4f01f" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:54.332034 24857 raft_consensus.cc:3037] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.331977 25261 consensus_queue.cc:260] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.332409 25261 raft_consensus.cc:3037] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.335239 24857 raft_consensus.cc:2454] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.335736 24926 leader_election.cc:302] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:54.336325 27286 raft_consensus.cc:2781] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:54.336490 27286 raft_consensus.cc:686] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.336743 27286 consensus_queue.cc:237] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.338696 25261 raft_consensus.cc:2454] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 2.
I20240502 21:29:54.341195 24703 catalog_manager.cc:5484] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 1 to 2, leader changed from 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 2 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } }
I20240502 21:29:54.364878 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f801dcce3bfc42bf89a41156826a9203"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:54.365129 24857 raft_consensus.cc:482] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.365262 24857 raft_consensus.cc:3037] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.367311 24857 raft_consensus.cc:504] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.368074 24857 leader_election.cc:288] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 2 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:54.368731 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f801dcce3bfc42bf89a41156826a9203" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.368943 25123 raft_consensus.cc:3037] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.368909 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f801dcce3bfc42bf89a41156826a9203" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:54.369175 24992 raft_consensus.cc:3032] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:54.369303 24992 raft_consensus.cc:729] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.369724 24992 consensus_queue.cc:260] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.370438 24992 raft_consensus.cc:3037] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.371528 25123 raft_consensus.cc:2454] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 2.
I20240502 21:29:54.373373 24992 raft_consensus.cc:2454] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 2.
I20240502 21:29:54.373987 24789 leader_election.cc:302] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:54.374441 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3c1ce7cf99164a9a848cbb22ad2052ea"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:54.374538 26556 raft_consensus.cc:2781] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:54.374647 24857 raft_consensus.cc:482] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.374770 24857 raft_consensus.cc:3037] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.375085 26556 raft_consensus.cc:686] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [term 2 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.375455 26556 consensus_queue.cc:237] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.377537 24857 raft_consensus.cc:504] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.378104 24857 leader_election.cc:288] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 3 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:54.379061 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1ce7cf99164a9a848cbb22ad2052ea" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.379065 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1ce7cf99164a9a848cbb22ad2052ea" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:54.379287 25123 raft_consensus.cc:3037] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.379289 24992 raft_consensus.cc:3032] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:54.379097 24703 catalog_manager.cc:5484] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 1 to 2, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 2 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:54.379418 24992 raft_consensus.cc:729] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.379791 24992 consensus_queue.cc:260] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.380173 24992 raft_consensus.cc:3037] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.381896 25123 raft_consensus.cc:2454] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 3.
I20240502 21:29:54.382143 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "083a9351f8ee4a44a86cb2d42f1ec558"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:54.382380 24857 raft_consensus.cc:482] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.382639 24857 raft_consensus.cc:3037] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.382602 24789 leader_election.cc:302] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:54.384980 26556 raft_consensus.cc:2781] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:54.385129 26556 raft_consensus.cc:686] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.385416 26556 consensus_queue.cc:237] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.385850 24857 raft_consensus.cc:504] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.386598 24857 leader_election.cc:288] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 2 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:54.387089 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "083a9351f8ee4a44a86cb2d42f1ec558" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.387214 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "083a9351f8ee4a44a86cb2d42f1ec558" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:54.387384 25123 raft_consensus.cc:3037] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.387519 25261 raft_consensus.cc:3032] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:54.387663 25261 raft_consensus.cc:729] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:54.387943 25261 consensus_queue.cc:260] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.388355 25261 raft_consensus.cc:3037] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.391410 25261 raft_consensus.cc:2454] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 2.
I20240502 21:29:54.391467 25123 raft_consensus.cc:2454] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 2.
I20240502 21:29:54.392184 24789 leader_election.cc:302] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:54.392515 27306 raft_consensus.cc:2781] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:54.392732 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f62981fdfb2349bd9fbe0e85b77ae492"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:54.393018 24857 raft_consensus.cc:482] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.393230 24857 raft_consensus.cc:3037] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.393266 24992 raft_consensus.cc:2396] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:54.398336 24857 raft_consensus.cc:504] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.400151 24857 leader_election.cc:288] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 3 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:54.403107 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62981fdfb2349bd9fbe0e85b77ae492" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.403363 25123 raft_consensus.cc:3037] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.403693 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bdecfa03390e405ca4925f8d1a40e0ca"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:54.403975 24857 raft_consensus.cc:482] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.404268 24857 raft_consensus.cc:3037] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.406495 25123 raft_consensus.cc:2454] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 3.
I20240502 21:29:54.407109 24789 leader_election.cc:302] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:54.408093 24857 raft_consensus.cc:504] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.408303 27309 raft_consensus.cc:2781] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:54.408440 27309 raft_consensus.cc:686] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.408749 27309 consensus_queue.cc:237] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.409526 27306 raft_consensus.cc:686] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [term 2 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.409487 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdecfa03390e405ca4925f8d1a40e0ca" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:54.409855 25261 raft_consensus.cc:3037] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.409857 27306 consensus_queue.cc:237] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.413291 25261 raft_consensus.cc:2454] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 3.
I20240502 21:29:54.413753 24790 leader_election.cc:302] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:54.411679 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62981fdfb2349bd9fbe0e85b77ae492" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:54.414186 24992 raft_consensus.cc:3032] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:54.414307 24992 raft_consensus.cc:729] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.414230 24857 leader_election.cc:288] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 3 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:54.414578 24992 consensus_queue.cc:260] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.414924 24992 raft_consensus.cc:3037] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.415772 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdecfa03390e405ca4925f8d1a40e0ca" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:54.416039 24991 raft_consensus.cc:3032] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:54.416183 24991 raft_consensus.cc:729] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.416502 24991 consensus_queue.cc:260] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.416898 24991 raft_consensus.cc:3037] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.418314 24992 raft_consensus.cc:2396] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:54.421742 24991 raft_consensus.cc:2396] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:54.422235 27310 raft_consensus.cc:2781] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:54.422464 27310 raft_consensus.cc:686] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.422971 27310 consensus_queue.cc:237] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.428938 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5360f56e93ef486b94bebcddf2edfaaa"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:54.429169 24857 raft_consensus.cc:482] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.429298 24857 raft_consensus.cc:3037] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.431054 24703 catalog_manager.cc:5484] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 2 to 3, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 3 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:54.433135 24857 raft_consensus.cc:504] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.433971 24857 leader_election.cc:288] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 2 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:54.434681 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5360f56e93ef486b94bebcddf2edfaaa" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:54.434906 25261 raft_consensus.cc:3032] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:54.434998 25261 raft_consensus.cc:729] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:54.435081 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57b40e72e8324b4ba6febf5eda24d333"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:54.435235 24857 raft_consensus.cc:482] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.435364 24857 raft_consensus.cc:3037] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.435276 25261 consensus_queue.cc:260] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.435673 25261 raft_consensus.cc:3037] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.437294 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5360f56e93ef486b94bebcddf2edfaaa" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.437480 25123 raft_consensus.cc:3037] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.437959 24857 raft_consensus.cc:504] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.438418 25261 raft_consensus.cc:2454] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 2.
I20240502 21:29:54.438616 24857 leader_election.cc:288] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 2 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:54.439489 24790 leader_election.cc:302] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:54.439764 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b40e72e8324b4ba6febf5eda24d333" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:54.439968 25260 raft_consensus.cc:3037] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.440063 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "22e48679f46b4b72aa0f0736c76be6f1"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:54.440279 24857 raft_consensus.cc:465] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 21:29:54.441322 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3e32a87a85274194916bef376c7e558e"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:54.441530 24857 raft_consensus.cc:482] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.441664 24857 raft_consensus.cc:3037] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.443013 25260 raft_consensus.cc:2454] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 2.
I20240502 21:29:54.443838 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b40e72e8324b4ba6febf5eda24d333" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:54.444036 24991 raft_consensus.cc:3032] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:54.444133 24991 raft_consensus.cc:729] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.444358 24991 consensus_queue.cc:260] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.444728 24991 raft_consensus.cc:3037] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.444754 24857 raft_consensus.cc:504] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.445344 24857 leader_election.cc:288] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 3 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:54.445785 25122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e32a87a85274194916bef376c7e558e" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.446017 25122 raft_consensus.cc:3037] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.447306 24991 raft_consensus.cc:2454] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 2.
I20240502 21:29:54.448407 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e32a87a85274194916bef376c7e558e" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:54.448491 24789 leader_election.cc:302] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:54.448602 24991 raft_consensus.cc:3032] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:54.448697 24991 raft_consensus.cc:729] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.448923 27306 raft_consensus.cc:2781] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:54.448935 25122 raft_consensus.cc:2454] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 3.
I20240502 21:29:54.449111 27306 raft_consensus.cc:686] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [term 2 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.448973 24991 consensus_queue.cc:260] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.445137 25123 raft_consensus.cc:2454] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 2.
I20240502 21:29:54.449445 24991 raft_consensus.cc:3037] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.449364 27306 consensus_queue.cc:237] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.450129 24789 leader_election.cc:302] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:54.450920 27312 raft_consensus.cc:2781] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:54.449136 26556 raft_consensus.cc:2781] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:54.452191 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a3a378f7a44648c3accc9dcba847040d"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:54.452257 26556 raft_consensus.cc:686] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [term 2 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.452412 24857 raft_consensus.cc:482] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.452626 24857 raft_consensus.cc:3037] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.452622 26556 consensus_queue.cc:237] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.455050 24991 raft_consensus.cc:2396] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:54.455881 24857 raft_consensus.cc:504] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.456986 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a378f7a44648c3accc9dcba847040d" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.457217 25123 raft_consensus.cc:3037] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.457204 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a378f7a44648c3accc9dcba847040d" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:54.457454 24991 raft_consensus.cc:3032] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:54.457571 24991 raft_consensus.cc:729] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.457908 24991 consensus_queue.cc:260] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.458290 24991 raft_consensus.cc:3037] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.459887 25123 raft_consensus.cc:2454] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 3.
I20240502 21:29:54.456619 24857 leader_election.cc:288] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 3 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:54.460532 24789 leader_election.cc:302] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:54.460827 27309 raft_consensus.cc:2781] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:54.461079 27309 raft_consensus.cc:686] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.461190 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a851d0bede1444c8a96ea45ea948bd59"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:54.461403 24857 raft_consensus.cc:482] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.461513 24857 raft_consensus.cc:3037] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.461436 27309 consensus_queue.cc:237] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.462813 24991 raft_consensus.cc:2396] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:54.466426 27312 raft_consensus.cc:686] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.466812 27312 consensus_queue.cc:237] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.468358 24857 raft_consensus.cc:504] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.470135 24857 leader_election.cc:288] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 3 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:54.470418 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851d0bede1444c8a96ea45ea948bd59" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:54.470597 24991 raft_consensus.cc:3032] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:54.470702 24991 raft_consensus.cc:729] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.470724 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851d0bede1444c8a96ea45ea948bd59" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:54.470995 25260 raft_consensus.cc:3037] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.470950 24991 consensus_queue.cc:260] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.471197 24991 raft_consensus.cc:3037] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.473037 24991 raft_consensus.cc:2396] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:54.473942 25260 raft_consensus.cc:2454] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 3.
I20240502 21:29:54.474354 24790 leader_election.cc:302] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters: 64d97dd1674d43ccadd7b4a38eeff557
I20240502 21:29:54.474629 27306 raft_consensus.cc:2781] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:54.474818 27306 raft_consensus.cc:686] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.475056 27306 consensus_queue.cc:237] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.476015 24703 catalog_manager.cc:5484] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 1 to 2, leader changed from 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 2 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } }
I20240502 21:29:54.476665 24703 catalog_manager.cc:5484] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 2 to 3, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 3 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } }
I20240502 21:29:54.477133 24703 catalog_manager.cc:5484] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 2 to 3, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 3 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } }
I20240502 21:29:54.477627 24703 catalog_manager.cc:5484] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 1 to 2, leader changed from 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 2 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:54.479722 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "175511733b714d2ea3cf877a0e5df418"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:54.479920 24857 raft_consensus.cc:482] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.480060 24857 raft_consensus.cc:3037] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.482684 24857 raft_consensus.cc:504] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.484117 24857 leader_election.cc:288] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 2 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:54.484303 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175511733b714d2ea3cf877a0e5df418" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:54.484526 24991 raft_consensus.cc:3032] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:54.484637 24991 raft_consensus.cc:729] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.484885 24991 consensus_queue.cc:260] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.485205 24991 raft_consensus.cc:3037] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.487025 24991 raft_consensus.cc:2454] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 2.
I20240502 21:29:54.487433 24789 leader_election.cc:302] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:54.487738 27306 raft_consensus.cc:2781] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:54.488081 27306 raft_consensus.cc:686] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b [term 2 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.488471 27306 consensus_queue.cc:237] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.488575 25122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175511733b714d2ea3cf877a0e5df418" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.488829 25122 raft_consensus.cc:3037] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.492792 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e3d3eab014a466fa4a421f7d9d20fba"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:54.493350 24857 raft_consensus.cc:482] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.493543 24857 raft_consensus.cc:3037] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.493527 25122 raft_consensus.cc:2454] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 2.
I20240502 21:29:54.495391 24857 raft_consensus.cc:504] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.496042 24857 leader_election.cc:288] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 3 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:54.496663 25122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3d3eab014a466fa4a421f7d9d20fba" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.496668 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3d3eab014a466fa4a421f7d9d20fba" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:54.496872 24991 raft_consensus.cc:3032] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:54.496959 24991 raft_consensus.cc:729] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.497135 25122 raft_consensus.cc:3037] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.497284 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b4874706759a403992f9662c84a62b91"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:54.497222 24991 consensus_queue.cc:260] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.497434 24857 raft_consensus.cc:482] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.497546 24857 raft_consensus.cc:3037] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.497545 24991 raft_consensus.cc:3037] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.500116 25122 raft_consensus.cc:2454] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 3.
I20240502 21:29:54.500315 24857 raft_consensus.cc:504] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.500556 24789 leader_election.cc:302] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:54.500608 24991 raft_consensus.cc:2396] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:54.500990 27306 raft_consensus.cc:2781] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:54.501147 27306 raft_consensus.cc:686] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.501322 24857 leader_election.cc:288] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 2 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:54.501885 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4874706759a403992f9662c84a62b91" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:54.502132 25260 raft_consensus.cc:3032] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:54.502110 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4874706759a403992f9662c84a62b91" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.502132 27306 consensus_queue.cc:237] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.502265 25260 raft_consensus.cc:729] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:54.502310 25123 raft_consensus.cc:3037] T b4874706759a403992f9662c84a62b91 P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.502543 25260 consensus_queue.cc:260] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.502939 25260 raft_consensus.cc:3037] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.504961 25123 raft_consensus.cc:2454] T b4874706759a403992f9662c84a62b91 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 2.
I20240502 21:29:54.505681 24789 leader_election.cc:302] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:54.505726 25260 raft_consensus.cc:2454] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 2.
I20240502 21:29:54.505995 27317 raft_consensus.cc:2781] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:54.506661 27317 raft_consensus.cc:686] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [term 2 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.507054 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "acf184a70440492c80cfaea7d80323de"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:54.507252 24857 raft_consensus.cc:465] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 21:29:54.507294 27317 consensus_queue.cc:237] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.508195 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c574664a57444c95b70d754cc46e313e"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:54.508432 24857 raft_consensus.cc:482] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.508535 24857 raft_consensus.cc:3037] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.510983 24857 raft_consensus.cc:504] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.511508 24857 leader_election.cc:288] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 2 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:54.512084 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c574664a57444c95b70d754cc46e313e" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:54.512316 24991 raft_consensus.cc:3032] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:54.512440 24991 raft_consensus.cc:729] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.512353 25122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c574664a57444c95b70d754cc46e313e" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.512228 24703 catalog_manager.cc:5484] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 1 to 2, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 2 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:54.512758 25122 raft_consensus.cc:3037] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.512717 24991 consensus_queue.cc:260] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.512827 24703 catalog_manager.cc:5484] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 2 to 3, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 3 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:54.513134 24991 raft_consensus.cc:3037] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.513281 24703 catalog_manager.cc:5484] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 2 to 3, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 3 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:54.513806 24703 catalog_manager.cc:5484] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 1 to 2, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 2 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:54.515091 24703 catalog_manager.cc:5484] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 2 to 3, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 3 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } }
I20240502 21:29:54.515969 24991 raft_consensus.cc:2454] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 2.
I20240502 21:29:54.515969 25122 raft_consensus.cc:2454] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 2.
I20240502 21:29:54.516383 24789 leader_election.cc:302] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:54.516571 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e145bbec053d4028ad7bd38d25dd4315"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:54.516780 27306 raft_consensus.cc:2781] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:54.517038 27306 raft_consensus.cc:686] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b [term 2 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.517334 27306 consensus_queue.cc:237] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.516798 24857 raft_consensus.cc:482] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.519039 24857 raft_consensus.cc:3037] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.522130 24857 raft_consensus.cc:504] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.522672 24857 leader_election.cc:288] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 2 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:54.523223 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e145bbec053d4028ad7bd38d25dd4315" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:54.523434 25260 raft_consensus.cc:3032] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:54.523550 25260 raft_consensus.cc:729] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:54.523886 25260 consensus_queue.cc:260] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.524294 25260 raft_consensus.cc:3037] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.525525 24703 catalog_manager.cc:5484] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 2 to 3, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 3 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:54.525971 24703 catalog_manager.cc:5484] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 1 to 2, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 2 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:54.526407 24703 catalog_manager.cc:5484] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 1 to 2, leader changed from 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 2 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:54.527065 25260 raft_consensus.cc:2454] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 2.
I20240502 21:29:54.528846 24790 leader_election.cc:302] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:54.529145 27306 raft_consensus.cc:2781] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:54.529356 27306 raft_consensus.cc:686] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [term 2 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.529646 27306 consensus_queue.cc:237] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.530761 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fed2d260d00d470e95e9a62cc71ba94f"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:54.530968 24857 raft_consensus.cc:482] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.531082 24857 raft_consensus.cc:3037] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.530977 25122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e145bbec053d4028ad7bd38d25dd4315" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.531343 25122 raft_consensus.cc:3037] T e145bbec053d4028ad7bd38d25dd4315 P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.533792 24857 raft_consensus.cc:504] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.534814 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fed2d260d00d470e95e9a62cc71ba94f" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:54.534941 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fed2d260d00d470e95e9a62cc71ba94f" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:54.535141 24991 raft_consensus.cc:3032] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:54.535187 25260 raft_consensus.cc:3037] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.535270 24991 raft_consensus.cc:729] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.535542 24991 consensus_queue.cc:260] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.535910 24991 raft_consensus.cc:3037] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.538159 25260 raft_consensus.cc:2454] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 3.
I20240502 21:29:54.538621 24790 leader_election.cc:302] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:54.538756 24991 raft_consensus.cc:2396] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:54.538914 27317 raft_consensus.cc:2781] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:54.539110 27317 raft_consensus.cc:686] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.539115 25122 raft_consensus.cc:2454] T e145bbec053d4028ad7bd38d25dd4315 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 2.
I20240502 21:29:54.539387 27317 consensus_queue.cc:237] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.542524 24857 leader_election.cc:288] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 3 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:54.543006 24703 catalog_manager.cc:5484] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 1 to 2, leader changed from 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 2 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:54.543866 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "89b06cd6b69d41d0b49ad40f605a3e16"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:54.544049 24857 raft_consensus.cc:482] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.544180 24857 raft_consensus.cc:3037] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.546639 24857 raft_consensus.cc:504] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.547410 24857 leader_election.cc:288] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 2 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:54.547672 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b06cd6b69d41d0b49ad40f605a3e16" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:54.547917 25260 raft_consensus.cc:3032] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:54.548004 25260 raft_consensus.cc:729] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:54.548372 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8423ff21d93741cdab3093ea454bb532"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:54.548328 25260 consensus_queue.cc:260] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.548574 24857 raft_consensus.cc:482] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.548689 24857 raft_consensus.cc:3037] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.548761 25260 raft_consensus.cc:3037] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.550603 24857 raft_consensus.cc:504] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.551172 24857 leader_election.cc:288] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 3 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:54.551784 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8423ff21d93741cdab3093ea454bb532" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:54.552022 24991 raft_consensus.cc:3032] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:54.552042 25122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b06cd6b69d41d0b49ad40f605a3e16" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.552134 24991 raft_consensus.cc:729] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.552277 25122 raft_consensus.cc:3037] T 89b06cd6b69d41d0b49ad40f605a3e16 P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.552384 24991 consensus_queue.cc:260] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.552726 24991 raft_consensus.cc:3037] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.554581 24991 raft_consensus.cc:2396] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:54.555020 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "986c305824414299ac30f1e50f514c8a"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:54.555187 24857 raft_consensus.cc:482] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.555224 25122 raft_consensus.cc:2454] T 89b06cd6b69d41d0b49ad40f605a3e16 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 2.
I20240502 21:29:54.555305 24857 raft_consensus.cc:3037] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.556020 24789 leader_election.cc:302] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:54.556315 27306 raft_consensus.cc:2781] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:54.556468 27306 raft_consensus.cc:686] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [term 2 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.551548 25260 raft_consensus.cc:2454] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 2.
I20240502 21:29:54.556702 27306 consensus_queue.cc:237] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.557397 24703 catalog_manager.cc:5484] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 2 to 3, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 3 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:54.552632 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8423ff21d93741cdab3093ea454bb532" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:54.558725 25261 raft_consensus.cc:3037] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.558835 24857 raft_consensus.cc:504] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.559880 24857 leader_election.cc:288] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 3 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:54.560437 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c305824414299ac30f1e50f514c8a" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:54.560609 24991 raft_consensus.cc:3032] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:54.560694 24991 raft_consensus.cc:729] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.560964 24991 consensus_queue.cc:260] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.561244 24991 raft_consensus.cc:3037] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.561272 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c305824414299ac30f1e50f514c8a" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:54.561471 25260 raft_consensus.cc:3037] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.561859 25261 raft_consensus.cc:2454] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 3.
I20240502 21:29:54.564020 24991 raft_consensus.cc:2396] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:54.568646 24790 leader_election.cc:302] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters: 64d97dd1674d43ccadd7b4a38eeff557
I20240502 21:29:54.569170 25260 raft_consensus.cc:2454] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 3.
I20240502 21:29:54.569532 27317 raft_consensus.cc:2781] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:54.569730 27317 raft_consensus.cc:686] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.570099 27317 consensus_queue.cc:237] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.574568 24790 leader_election.cc:302] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters: 64d97dd1674d43ccadd7b4a38eeff557
I20240502 21:29:54.574841 27306 raft_consensus.cc:2781] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:54.575048 27306 raft_consensus.cc:686] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.575304 27306 consensus_queue.cc:237] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.582470 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "48e7c6534b544336ae216d3200f4f01f"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:54.582688 24857 raft_consensus.cc:482] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.582824 24857 raft_consensus.cc:3037] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.586266 24703 catalog_manager.cc:5484] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 2 to 3, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 3 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:54.586851 24703 catalog_manager.cc:5484] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 1 to 2, leader changed from 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 2 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:54.587273 24703 catalog_manager.cc:5484] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 2 to 3, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 3 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:54.588605 27134 heartbeater.cc:503] Master 127.29.82.126:45283 was elected leader, sending a full tablet report...
I20240502 21:29:54.586452 24857 raft_consensus.cc:504] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.591354 24857 leader_election.cc:288] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 3 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:54.591403 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e7c6534b544336ae216d3200f4f01f" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:54.591665 25260 raft_consensus.cc:3037] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.591816 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e7c6534b544336ae216d3200f4f01f" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:54.592064 24991 raft_consensus.cc:3032] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:54.592180 24991 raft_consensus.cc:729] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.592430 24991 consensus_queue.cc:260] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.592756 24991 raft_consensus.cc:3037] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.594465 25260 raft_consensus.cc:2454] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 3.
I20240502 21:29:54.594986 24790 leader_election.cc:302] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:54.595350 27317 raft_consensus.cc:2781] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:54.595535 27317 raft_consensus.cc:686] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.595911 27317 consensus_queue.cc:237] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.597424 25260 raft_consensus.cc:1261] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Refusing update from remote peer 64d97dd1674d43ccadd7b4a38eeff557: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:29:54.597743 24991 raft_consensus.cc:2396] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:54.599130 27286 consensus_queue.cc:1035] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:54.605772 24703 catalog_manager.cc:5484] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 2 to 3, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 3 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } }
I20240502 21:29:54.618212 25122 raft_consensus.cc:1261] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Refusing update from remote peer 64d97dd1674d43ccadd7b4a38eeff557: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:29:54.619216 26535 consensus_queue.cc:1035] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:54.653620 25122 raft_consensus.cc:1261] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Refusing update from remote peer 64d97dd1674d43ccadd7b4a38eeff557: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:29:54.654210 26535 consensus_queue.cc:1035] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:54.665915 25123 raft_consensus.cc:1261] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Refusing update from remote peer 64d97dd1674d43ccadd7b4a38eeff557: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:29:54.666716 26535 consensus_queue.cc:1035] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:54.675659 25260 raft_consensus.cc:1261] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Refusing update from remote peer 64d97dd1674d43ccadd7b4a38eeff557: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:29:54.676558 27151 consensus_queue.cc:1035] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:54.689843 25260 raft_consensus.cc:1261] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Refusing update from remote peer 64d97dd1674d43ccadd7b4a38eeff557: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:29:54.692196 27151 consensus_queue.cc:1035] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:54.696215 25123 raft_consensus.cc:1261] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Refusing update from remote peer 64d97dd1674d43ccadd7b4a38eeff557: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:29:54.696936 27286 consensus_queue.cc:1035] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:54.702328 25260 raft_consensus.cc:1261] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Refusing update from remote peer 64d97dd1674d43ccadd7b4a38eeff557: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:29:54.703084 27286 consensus_queue.cc:1035] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:54.715003 25260 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "87459ddf48034b28b7b3e0d0d7de91f1"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:54.715296 25260 raft_consensus.cc:482] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.715399 25260 raft_consensus.cc:3037] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.717491 25260 raft_consensus.cc:504] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.718111 25260 leader_election.cc:288] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 2 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:54.718580 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87459ddf48034b28b7b3e0d0d7de91f1" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.718837 25123 raft_consensus.cc:3037] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.719209 25260 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4127d80443f1499ab9a30935bbd4ca8e"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:54.719338 25260 raft_consensus.cc:482] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.719424 25260 raft_consensus.cc:3037] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.721341 25260 raft_consensus.cc:504] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.721666 25123 raft_consensus.cc:2454] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 2.
I20240502 21:29:54.722096 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87459ddf48034b28b7b3e0d0d7de91f1" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:54.722303 24991 raft_consensus.cc:3032] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:54.722348 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4127d80443f1499ab9a30935bbd4ca8e" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:54.722404 24991 raft_consensus.cc:729] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.722504 24992 raft_consensus.cc:3032] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:54.722604 24992 raft_consensus.cc:729] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.722702 24991 consensus_queue.cc:260] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.722856 24992 consensus_queue.cc:260] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.723094 24991 raft_consensus.cc:3037] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.723220 24992 raft_consensus.cc:3037] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.723682 25260 leader_election.cc:288] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 3 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:54.724884 25261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "22e48679f46b4b72aa0f0736c76be6f1"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:54.725052 25261 raft_consensus.cc:482] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.725170 25261 raft_consensus.cc:3037] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.725602 24992 raft_consensus.cc:2454] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 3.
I20240502 21:29:54.725968 25191 leader_election.cc:302] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:54.726104 24991 raft_consensus.cc:2454] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 2.
I20240502 21:29:54.726280 26870 raft_consensus.cc:2781] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:54.726622 25191 leader_election.cc:302] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:54.727284 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4127d80443f1499ab9a30935bbd4ca8e" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.727532 25123 raft_consensus.cc:3037] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.727814 25261 raft_consensus.cc:504] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.728477 25261 leader_election.cc:288] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 2 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:54.729517 25260 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d643c15afbec4933a0d0a145f17f703c"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:54.729710 25260 raft_consensus.cc:482] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.729826 25260 raft_consensus.cc:3037] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.731524 25261 raft_consensus.cc:1261] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Refusing update from remote peer 64d97dd1674d43ccadd7b4a38eeff557: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:29:54.732167 27286 consensus_queue.cc:1035] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:54.732657 25260 raft_consensus.cc:504] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.733323 25260 leader_election.cc:288] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 3 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:54.734192 25123 raft_consensus.cc:2454] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 3.
I20240502 21:29:54.734814 26870 raft_consensus.cc:686] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:54.735389 27359 raft_consensus.cc:2781] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:54.735414 25258 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "083a9351f8ee4a44a86cb2d42f1ec558"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:54.735615 25258 raft_consensus.cc:482] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.735304 26870 consensus_queue.cc:237] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.735771 25258 raft_consensus.cc:3037] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.738085 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22e48679f46b4b72aa0f0736c76be6f1" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:54.738260 24857 raft_consensus.cc:3032] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:54.738332 24857 raft_consensus.cc:729] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.738570 24857 consensus_queue.cc:260] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.738577 25258 raft_consensus.cc:504] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.738860 24857 raft_consensus.cc:3037] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.738787 24703 catalog_manager.cc:5484] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 2 to 3, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 3 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:54.739173 25258 leader_election.cc:288] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 3 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:54.740218 25258 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bdecfa03390e405ca4925f8d1a40e0ca"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:54.740464 25258 raft_consensus.cc:482] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.740636 25258 raft_consensus.cc:3037] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:54.741365 25122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "083a9351f8ee4a44a86cb2d42f1ec558" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.741591 25122 raft_consensus.cc:3037] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.741672 24857 raft_consensus.cc:2454] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 2.
I20240502 21:29:54.741910 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22e48679f46b4b72aa0f0736c76be6f1" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.742110 25123 raft_consensus.cc:3037] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.742796 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d643c15afbec4933a0d0a145f17f703c" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:54.743044 24991 raft_consensus.cc:3032] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:54.743157 24991 raft_consensus.cc:729] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.743465 24991 consensus_queue.cc:260] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.743813 24991 raft_consensus.cc:3037] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.744623 25122 raft_consensus.cc:2454] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 3.
I20240502 21:29:54.744904 25123 raft_consensus.cc:2454] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 2.
I20240502 21:29:54.745242 25124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d643c15afbec4933a0d0a145f17f703c" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.745419 25124 raft_consensus.cc:3037] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.745645 24991 raft_consensus.cc:2454] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 3.
I20240502 21:29:54.746022 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "083a9351f8ee4a44a86cb2d42f1ec558" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:54.746171 24857 raft_consensus.cc:3032] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:54.746237 24857 raft_consensus.cc:729] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.746448 24857 consensus_queue.cc:260] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.746646 24857 raft_consensus.cc:3037] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.747285 25258 raft_consensus.cc:504] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.747764 25191 leader_election.cc:302] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:54.747972 25258 leader_election.cc:288] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 4 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:54.748016 25124 raft_consensus.cc:2454] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 3.
I20240502 21:29:54.748134 25191 leader_election.cc:302] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:54.748873 25258 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5360f56e93ef486b94bebcddf2edfaaa"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:54.748922 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdecfa03390e405ca4925f8d1a40e0ca" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:54.749034 25258 raft_consensus.cc:482] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.749114 24991 raft_consensus.cc:3037] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:54.749156 25258 raft_consensus.cc:3037] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.749219 26870 raft_consensus.cc:2781] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:54.749373 26870 raft_consensus.cc:686] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:54.749424 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdecfa03390e405ca4925f8d1a40e0ca" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:54.749634 24856 raft_consensus.cc:3032] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:29:54.749706 24856 raft_consensus.cc:729] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.749691 26870 consensus_queue.cc:237] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.749958 24856 consensus_queue.cc:260] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.750226 24856 raft_consensus.cc:3037] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:54.750612 26870 raft_consensus.cc:2781] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:54.750731 26870 raft_consensus.cc:686] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [term 2 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:54.750963 26870 consensus_queue.cc:237] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.752023 24991 raft_consensus.cc:2396] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:54.752786 24857 raft_consensus.cc:2396] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:54.753394 25191 leader_election.cc:302] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced; no voters: a171ad73f86643afb1444248c0d0bc8b
I20240502 21:29:54.753916 25258 raft_consensus.cc:504] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.755652 24856 raft_consensus.cc:2396] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:54.755990 26870 raft_consensus.cc:2781] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:54.756354 25124 raft_consensus.cc:1261] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Refusing update from remote peer 64d97dd1674d43ccadd7b4a38eeff557: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:29:54.756224 25258 leader_election.cc:288] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 3 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:54.756986 26870 raft_consensus.cc:686] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:54.756927 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5360f56e93ef486b94bebcddf2edfaaa" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:54.756940 27286 consensus_queue.cc:1035] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:54.757128 24857 raft_consensus.cc:3032] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:54.757239 24857 raft_consensus.cc:729] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.757347 25260 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57b40e72e8324b4ba6febf5eda24d333"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:54.757536 25260 raft_consensus.cc:482] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.757512 24857 consensus_queue.cc:260] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.757668 25260 raft_consensus.cc:3037] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.757830 24857 raft_consensus.cc:3037] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.757853 26870 consensus_queue.cc:237] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.760903 24857 raft_consensus.cc:2396] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:54.761816 25124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5360f56e93ef486b94bebcddf2edfaaa" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.762120 25124 raft_consensus.cc:3037] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.763653 25260 raft_consensus.cc:504] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.764187 25191 leader_election.cc:302] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c; no voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b
I20240502 21:29:54.764426 25260 leader_election.cc:288] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 3 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:54.764568 27358 raft_consensus.cc:2726] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:29:54.765316 25258 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "69afb121cb564de3a9bec5e3f1cc0492"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:54.765518 25258 raft_consensus.cc:482] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.765640 25258 raft_consensus.cc:3037] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.766108 25124 raft_consensus.cc:2454] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 3.
I20240502 21:29:54.767199 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b40e72e8324b4ba6febf5eda24d333" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:54.767439 24856 raft_consensus.cc:3032] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:54.767601 24856 raft_consensus.cc:729] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.767597 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b40e72e8324b4ba6febf5eda24d333" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:54.767846 24991 raft_consensus.cc:3037] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.768029 24856 consensus_queue.cc:260] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.768458 24856 raft_consensus.cc:3037] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.768582 25258 raft_consensus.cc:504] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.769163 25258 leader_election.cc:288] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 3 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:54.770104 25261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "89c4eee02b444dc188172fd8c7b411bb"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:54.770282 25261 raft_consensus.cc:482] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.770411 25261 raft_consensus.cc:3037] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.771106 24991 raft_consensus.cc:2454] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 3.
I20240502 21:29:54.771660 25191 leader_election.cc:302] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced; no voters: a171ad73f86643afb1444248c0d0bc8b
I20240502 21:29:54.772660 25122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69afb121cb564de3a9bec5e3f1cc0492" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.772900 25122 raft_consensus.cc:3037] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.773061 25261 raft_consensus.cc:504] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.774365 25261 leader_election.cc:288] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 3 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:54.776212 25122 raft_consensus.cc:2454] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 3.
I20240502 21:29:54.776793 24856 raft_consensus.cc:2396] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:54.777233 27358 raft_consensus.cc:2781] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:54.777230 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69afb121cb564de3a9bec5e3f1cc0492" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:54.777460 24991 raft_consensus.cc:3032] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:54.777496 27358 raft_consensus.cc:686] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:54.777582 24991 raft_consensus.cc:729] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.777886 27358 consensus_queue.cc:237] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.777922 24991 consensus_queue.cc:260] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.778805 24991 raft_consensus.cc:3037] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.779690 25260 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b4874706759a403992f9662c84a62b91"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:54.779872 25260 raft_consensus.cc:482] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.779979 25260 raft_consensus.cc:3037] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.781386 25261 raft_consensus.cc:1261] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Refusing update from remote peer 64d97dd1674d43ccadd7b4a38eeff557: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:29:54.781906 27286 consensus_queue.cc:1035] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:54.782307 24991 raft_consensus.cc:2454] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 3.
I20240502 21:29:54.783022 25191 leader_election.cc:302] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:54.793089 25260 raft_consensus.cc:504] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.801028 25260 leader_election.cc:288] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 3 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:54.803853 27365 raft_consensus.cc:2781] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:54.804103 25260 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "acf184a70440492c80cfaea7d80323de"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:54.804073 27365 raft_consensus.cc:686] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:54.804879 25260 raft_consensus.cc:482] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.805310 25260 raft_consensus.cc:3037] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.805521 25191 leader_election.cc:302] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:54.806203 27359 raft_consensus.cc:686] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [term 2 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:54.807103 27361 raft_consensus.cc:2781] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:54.807271 27361 raft_consensus.cc:686] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:54.808377 25124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c4eee02b444dc188172fd8c7b411bb" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.808624 25124 raft_consensus.cc:3037] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.808915 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4874706759a403992f9662c84a62b91" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:54.809149 24856 raft_consensus.cc:3032] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:54.809223 24856 raft_consensus.cc:729] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.809541 24856 consensus_queue.cc:260] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.809862 24856 raft_consensus.cc:3037] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.811471 25124 raft_consensus.cc:2454] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 3.
I20240502 21:29:54.811794 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c4eee02b444dc188172fd8c7b411bb" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:54.811745 27359 consensus_queue.cc:237] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.811997 24991 raft_consensus.cc:3032] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:54.812134 24991 raft_consensus.cc:729] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.812390 24991 consensus_queue.cc:260] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.812798 24991 raft_consensus.cc:3037] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.813064 24856 raft_consensus.cc:2396] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:54.815101 24991 raft_consensus.cc:2454] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 3.
I20240502 21:29:54.815873 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4874706759a403992f9662c84a62b91" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.805896 27365 consensus_queue.cc:237] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.816089 25123 raft_consensus.cc:3037] T b4874706759a403992f9662c84a62b91 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.817349 27361 consensus_queue.cc:237] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.818753 25123 raft_consensus.cc:2454] T b4874706759a403992f9662c84a62b91 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 3.
I20240502 21:29:54.819046 25191 leader_election.cc:302] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:54.819639 25191 leader_election.cc:302] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced; no voters: a171ad73f86643afb1444248c0d0bc8b
I20240502 21:29:54.819705 25123 raft_consensus.cc:1261] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Refusing update from remote peer 64d97dd1674d43ccadd7b4a38eeff557: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:29:54.820354 27358 raft_consensus.cc:2781] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:54.820315 26535 consensus_queue.cc:1035] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:54.820488 27358 raft_consensus.cc:686] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:54.811537 25260 raft_consensus.cc:504] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.820734 27358 consensus_queue.cc:237] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.821344 25260 leader_election.cc:288] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 2 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:54.824234 25259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a851d0bede1444c8a96ea45ea948bd59"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:54.824437 25259 raft_consensus.cc:482] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.824584 25259 raft_consensus.cc:3037] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:54.830189 25259 raft_consensus.cc:504] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.831256 25259 leader_election.cc:288] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 4 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:54.832492 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf184a70440492c80cfaea7d80323de" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:54.832653 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf184a70440492c80cfaea7d80323de" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.832849 25123 raft_consensus.cc:3037] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.832865 24856 raft_consensus.cc:3032] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:54.833035 24856 raft_consensus.cc:729] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.834334 24856 consensus_queue.cc:260] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.834789 24856 raft_consensus.cc:3037] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.835485 25123 raft_consensus.cc:2454] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 2.
I20240502 21:29:54.836484 25260 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e145bbec053d4028ad7bd38d25dd4315"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:54.836687 25260 raft_consensus.cc:482] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.836808 25260 raft_consensus.cc:3037] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.838268 24856 raft_consensus.cc:2454] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 2.
I20240502 21:29:54.839097 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851d0bede1444c8a96ea45ea948bd59" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:54.839278 24991 raft_consensus.cc:3037] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:54.839509 25260 raft_consensus.cc:504] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.840131 25260 leader_election.cc:288] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 3 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:54.840157 25122 raft_consensus.cc:1261] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Refusing update from remote peer a171ad73f86643afb1444248c0d0bc8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:29:54.840869 27317 consensus_queue.cc:1035] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:54.842020 24991 raft_consensus.cc:2396] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:54.842754 25259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fed2d260d00d470e95e9a62cc71ba94f"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:54.842948 25259 raft_consensus.cc:482] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.843065 25259 raft_consensus.cc:3037] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:54.843081 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851d0bede1444c8a96ea45ea948bd59" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:54.843360 24856 raft_consensus.cc:3032] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:29:54.843480 24856 raft_consensus.cc:729] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.843801 24856 consensus_queue.cc:260] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.844059 26870 raft_consensus.cc:2781] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:54.844331 24703 catalog_manager.cc:5484] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 1 to 2, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 2 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:54.844969 25122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e145bbec053d4028ad7bd38d25dd4315" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.844727 26870 raft_consensus.cc:686] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:54.845189 25122 raft_consensus.cc:3037] T e145bbec053d4028ad7bd38d25dd4315 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.845621 24856 raft_consensus.cc:3037] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:54.845758 25191 leader_election.cc:302] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced; no voters:
I20240502 21:29:54.847096 27358 raft_consensus.cc:2781] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:54.847281 27358 raft_consensus.cc:686] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [term 2 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:54.847548 27358 consensus_queue.cc:237] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.847918 25122 raft_consensus.cc:2454] T e145bbec053d4028ad7bd38d25dd4315 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 3.
I20240502 21:29:54.844722 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e145bbec053d4028ad7bd38d25dd4315" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:54.848357 24857 raft_consensus.cc:3032] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:54.850196 25259 raft_consensus.cc:504] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.850889 25259 leader_election.cc:288] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 4 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:54.851917 25259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f25dfc6b0c04d878cd3db58d37cddec"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:54.852118 25259 raft_consensus.cc:482] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.852252 25259 raft_consensus.cc:3037] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.845927 26870 consensus_queue.cc:237] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.854075 25191 leader_election.cc:302] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced; no voters:
I20240502 21:29:54.854756 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fed2d260d00d470e95e9a62cc71ba94f" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:54.854961 24991 raft_consensus.cc:3037] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:54.855289 27358 raft_consensus.cc:2781] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:54.855473 27358 raft_consensus.cc:686] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:54.855780 27358 consensus_queue.cc:237] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.856709 24857 raft_consensus.cc:729] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.856972 24857 consensus_queue.cc:260] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.857242 24857 raft_consensus.cc:3037] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.857856 24991 raft_consensus.cc:2396] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:54.858184 24855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fed2d260d00d470e95e9a62cc71ba94f" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:54.858356 24855 raft_consensus.cc:3032] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:29:54.858441 24855 raft_consensus.cc:729] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.858664 24855 consensus_queue.cc:260] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.858912 24855 raft_consensus.cc:3037] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:54.859180 25259 raft_consensus.cc:504] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.859769 25259 leader_election.cc:288] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 3 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:54.861850 24857 raft_consensus.cc:2396] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:54.863922 25259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "69df123d92d74693b4654c6a809bbe89"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:54.864114 25259 raft_consensus.cc:482] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.864219 25259 raft_consensus.cc:3037] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.865199 24855 raft_consensus.cc:2396] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:54.866874 25259 raft_consensus.cc:504] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.867286 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f25dfc6b0c04d878cd3db58d37cddec" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:54.867542 24991 raft_consensus.cc:3032] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:54.867607 25259 leader_election.cc:288] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 3 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:54.867753 24991 raft_consensus.cc:729] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.868561 25259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "89b06cd6b69d41d0b49ad40f605a3e16"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:54.868816 25259 raft_consensus.cc:482] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.868945 25259 raft_consensus.cc:3037] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.871232 24703 catalog_manager.cc:5484] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 2 to 3, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 3 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:54.868312 24991 consensus_queue.cc:260] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.871887 25259 raft_consensus.cc:504] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.872207 24991 raft_consensus.cc:3037] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.872540 25259 leader_election.cc:288] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 3 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:54.873379 25259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "52153a02d80043f0a1840b0e5aadd9b3"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:54.873595 25259 raft_consensus.cc:482] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.873703 25259 raft_consensus.cc:3037] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.873852 24703 catalog_manager.cc:5484] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 2 to 3, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 3 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:54.874347 24703 catalog_manager.cc:5484] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 1 to 2, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 2 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } }
I20240502 21:29:54.875007 24703 catalog_manager.cc:5484] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 2 to 3, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 3 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:54.875476 24703 catalog_manager.cc:5484] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 2 to 3, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 3 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:54.876019 24703 catalog_manager.cc:5484] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 2 to 3, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 3 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } }
I20240502 21:29:54.876256 25259 raft_consensus.cc:504] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.876596 24703 catalog_manager.cc:5484] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 1 to 2, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 2 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:54.876854 25259 leader_election.cc:288] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 2 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:54.877132 24703 catalog_manager.cc:5484] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 2 to 3, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 3 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:54.877826 25259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "986c305824414299ac30f1e50f514c8a"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:54.878000 25259 raft_consensus.cc:482] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.878137 25259 raft_consensus.cc:3037] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:54.877826 24703 catalog_manager.cc:5484] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 2 to 3, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 3 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:54.880477 25259 raft_consensus.cc:504] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.881085 25259 leader_election.cc:288] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 4 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:54.882434 25259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8423ff21d93741cdab3093ea454bb532"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:54.882560 24992 raft_consensus.cc:1261] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Refusing update from remote peer a171ad73f86643afb1444248c0d0bc8b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 21:29:54.882645 25259 raft_consensus.cc:482] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.882748 24992 pending_rounds.cc:76] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557: Aborting all ops after (but not including) 3
I20240502 21:29:54.882824 25259 raft_consensus.cc:3037] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:54.882872 24992 pending_rounds.cc:97] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 21:29:54.883060 24992 raft_consensus.cc:2866] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:29:54.883982 27317 consensus_queue.cc:1035] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:54.885452 24991 raft_consensus.cc:2454] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 3.
I20240502 21:29:54.886440 25191 leader_election.cc:302] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c; no voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b
I20240502 21:29:54.886862 25191 leader_election.cc:302] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:54.886488 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f25dfc6b0c04d878cd3db58d37cddec" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.888860 25123 raft_consensus.cc:3037] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.889611 24856 raft_consensus.cc:2396] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:54.889905 27359 raft_consensus.cc:2726] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:29:54.890072 27359 raft_consensus.cc:2781] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:54.890194 27359 raft_consensus.cc:686] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:54.890473 27359 consensus_queue.cc:237] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.892508 25122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69df123d92d74693b4654c6a809bbe89" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.892481 24855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c305824414299ac30f1e50f514c8a" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:54.892745 24855 raft_consensus.cc:3032] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:29:54.892846 24855 raft_consensus.cc:729] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.892887 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c305824414299ac30f1e50f514c8a" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:54.892930 25121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b06cd6b69d41d0b49ad40f605a3e16" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.893076 24989 raft_consensus.cc:3037] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:54.893167 25121 raft_consensus.cc:3037] T 89b06cd6b69d41d0b49ad40f605a3e16 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.893038 25120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52153a02d80043f0a1840b0e5aadd9b3" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:54.893106 24855 consensus_queue.cc:260] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:54.893318 25120 raft_consensus.cc:3037] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.893416 24855 raft_consensus.cc:3037] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:54.896008 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69df123d92d74693b4654c6a809bbe89" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:54.896190 24992 raft_consensus.cc:3032] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:54.896317 24855 raft_consensus.cc:2396] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:54.896312 24992 raft_consensus.cc:729] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.896350 24989 raft_consensus.cc:2396] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:54.892201 25191 leader_election.cc:302] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c; no voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b
I20240502 21:29:54.896587 24992 consensus_queue.cc:260] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.896690 25120 raft_consensus.cc:2454] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 2.
I20240502 21:29:54.896946 24992 raft_consensus.cc:3037] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.897087 25191 leader_election.cc:302] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced; no voters:
I20240502 21:29:54.897367 27365 raft_consensus.cc:2726] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:29:54.897531 27365 raft_consensus.cc:2781] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:54.897642 27365 raft_consensus.cc:686] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [term 2 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:54.897881 27365 consensus_queue.cc:237] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.898520 25123 raft_consensus.cc:2454] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 3.
I20240502 21:29:54.885864 25259 raft_consensus.cc:504] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.901099 25121 raft_consensus.cc:2454] T 89b06cd6b69d41d0b49ad40f605a3e16 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 3.
I20240502 21:29:54.901309 25259 leader_election.cc:288] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 4 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:54.902338 24992 raft_consensus.cc:2454] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 3.
I20240502 21:29:54.902521 25259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "48e7c6534b544336ae216d3200f4f01f"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:54.902796 25259 raft_consensus.cc:482] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:54.902940 25259 raft_consensus.cc:3037] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:54.905051 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52153a02d80043f0a1840b0e5aadd9b3" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:54.905902 25259 raft_consensus.cc:504] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.906559 25259 leader_election.cc:288] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 4 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:54.908489 24991 raft_consensus.cc:3032] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:54.909129 24991 raft_consensus.cc:729] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:54.907716 24989 raft_consensus.cc:1261] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Refusing update from remote peer a171ad73f86643afb1444248c0d0bc8b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 21:29:54.910176 24989 pending_rounds.cc:76] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557: Aborting all ops after (but not including) 3
I20240502 21:29:54.910259 24989 pending_rounds.cc:97] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 21:29:54.910365 24989 raft_consensus.cc:2866] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:29:54.911109 27312 consensus_queue.cc:1035] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:54.908264 24992 raft_consensus.cc:1261] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Refusing update from remote peer a171ad73f86643afb1444248c0d0bc8b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 21:29:54.918363 24992 pending_rounds.cc:76] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557: Aborting all ops after (but not including) 3
I20240502 21:29:54.918493 24992 pending_rounds.cc:97] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 21:29:54.918589 24992 raft_consensus.cc:2866] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:29:54.919123 24703 catalog_manager.cc:5484] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 2 to 3, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 3 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:54.919883 27309 consensus_queue.cc:1035] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:54.921255 24855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8423ff21d93741cdab3093ea454bb532" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:54.921649 24855 raft_consensus.cc:3032] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:29:54.921844 24855 raft_consensus.cc:729] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.922355 24855 consensus_queue.cc:260] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.925222 24855 raft_consensus.cc:3037] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:54.926576 24991 consensus_queue.cc:260] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:54.927084 24991 raft_consensus.cc:3037] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:54.931253 24989 raft_consensus.cc:1261] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Refusing update from remote peer a171ad73f86643afb1444248c0d0bc8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:29:54.931959 27309 consensus_queue.cc:1035] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:54.933295 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e7c6534b544336ae216d3200f4f01f" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:54.933306 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e7c6534b544336ae216d3200f4f01f" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:54.933494 24990 raft_consensus.cc:3037] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:54.933483 24857 raft_consensus.cc:3032] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:29:54.933593 24857 raft_consensus.cc:729] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.933848 24857 consensus_queue.cc:260] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:54.934336 24857 raft_consensus.cc:3037] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:54.935982 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8423ff21d93741cdab3093ea454bb532" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:54.936149 24992 raft_consensus.cc:3037] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:54.936455 24990 raft_consensus.cc:2396] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:54.937875 24857 raft_consensus.cc:2396] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:54.940189 24989 raft_consensus.cc:1261] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Refusing update from remote peer a171ad73f86643afb1444248c0d0bc8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:29:54.940814 25124 raft_consensus.cc:1261] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Refusing update from remote peer a171ad73f86643afb1444248c0d0bc8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 21:29:54.940815 25191 leader_election.cc:302] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters: 5bf90e514781418da921c046aa5d9ced
I20240502 21:29:54.941586 27310 consensus_queue.cc:1035] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:54.943011 27310 consensus_queue.cc:1035] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:54.944916 24855 raft_consensus.cc:2396] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:54.945137 27365 raft_consensus.cc:2781] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:54.945303 27365 raft_consensus.cc:686] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:54.945571 27365 consensus_queue.cc:237] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.947832 25121 raft_consensus.cc:1261] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Refusing update from remote peer a171ad73f86643afb1444248c0d0bc8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:29:54.952283 25191 leader_election.cc:302] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c; no voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b
I20240502 21:29:54.952376 25124 raft_consensus.cc:1261] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Refusing update from remote peer a171ad73f86643afb1444248c0d0bc8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 21:29:54.948571 27309 consensus_queue.cc:1035] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:54.953068 24711 catalog_manager.cc:5484] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 2 to 3, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 3 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:54.953661 27310 consensus_queue.cc:1035] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:54.953698 24711 catalog_manager.cc:5484] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 2 to 3, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 3 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:54.954185 26870 raft_consensus.cc:2726] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:29:54.954550 25191 leader_election.cc:302] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c; no voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b
I20240502 21:29:54.954476 24711 catalog_manager.cc:5484] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 1 to 2, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 2 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:54.954813 26870 raft_consensus.cc:2726] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:29:54.955546 25191 leader_election.cc:302] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced; no voters:
I20240502 21:29:54.958493 26870 raft_consensus.cc:2781] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:54.958691 26870 raft_consensus.cc:686] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:54.959070 26870 consensus_queue.cc:237] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.969265 24711 catalog_manager.cc:5484] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 2 to 3, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 3 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:54.969975 24992 raft_consensus.cc:2396] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:54.970566 25191 leader_election.cc:302] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c; no voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b
I20240502 21:29:54.970947 27365 raft_consensus.cc:2726] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:29:54.971897 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b06cd6b69d41d0b49ad40f605a3e16" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:54.972921 24856 raft_consensus.cc:3032] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:54.973055 24856 raft_consensus.cc:729] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:54.973408 24856 consensus_queue.cc:260] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:54.973774 24856 raft_consensus.cc:3037] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:54.975261 25124 raft_consensus.cc:1261] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Refusing update from remote peer a171ad73f86643afb1444248c0d0bc8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:29:54.976261 27306 consensus_queue.cc:1035] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:54.976800 24991 raft_consensus.cc:2454] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 2.
I20240502 21:29:54.980829 25121 raft_consensus.cc:1261] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Refusing update from remote peer a171ad73f86643afb1444248c0d0bc8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 21:29:54.982468 27317 consensus_queue.cc:1035] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:54.990971 24991 raft_consensus.cc:1261] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Refusing update from remote peer a171ad73f86643afb1444248c0d0bc8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:29:54.991333 24992 raft_consensus.cc:1261] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Refusing update from remote peer a171ad73f86643afb1444248c0d0bc8b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 21:29:54.992100 24992 pending_rounds.cc:76] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557: Aborting all ops after (but not including) 3
I20240502 21:29:54.992178 24992 pending_rounds.cc:97] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 21:29:54.992074 27317 consensus_queue.cc:1035] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:54.992269 24992 raft_consensus.cc:2866] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:29:54.992820 27317 consensus_queue.cc:1035] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:54.995911 25124 raft_consensus.cc:1261] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Refusing update from remote peer a171ad73f86643afb1444248c0d0bc8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 21:29:54.996364 27317 consensus_queue.cc:1035] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:55.005995 24856 raft_consensus.cc:2396] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:55.018464 24991 raft_consensus.cc:1261] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Refusing update from remote peer a171ad73f86643afb1444248c0d0bc8b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 21:29:55.018615 24991 pending_rounds.cc:76] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557: Aborting all ops after (but not including) 3
I20240502 21:29:55.018682 24991 pending_rounds.cc:97] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 21:29:55.018762 24991 raft_consensus.cc:2866] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:29:55.019825 27306 consensus_queue.cc:1035] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:55.024483 25123 raft_consensus.cc:1261] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Refusing update from remote peer a171ad73f86643afb1444248c0d0bc8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 21:29:55.025220 26556 consensus_queue.cc:1035] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:55.057057 25123 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4127d80443f1499ab9a30935bbd4ca8e"
dest_uuid: "5bf90e514781418da921c046aa5d9ced"
from {username='slave'} at 127.0.0.1:52128
I20240502 21:29:55.057385 25123 raft_consensus.cc:482] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.057518 25123 raft_consensus.cc:3037] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.061312 25123 raft_consensus.cc:504] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.062783 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4127d80443f1499ab9a30935bbd4ca8e" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.063035 24991 raft_consensus.cc:3037] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.063426 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4127d80443f1499ab9a30935bbd4ca8e" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.063740 25259 raft_consensus.cc:3032] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:29:55.063866 25259 raft_consensus.cc:729] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:55.064252 25259 consensus_queue.cc:260] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.064687 25259 raft_consensus.cc:3037] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.065923 24991 raft_consensus.cc:2454] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 4.
I20240502 21:29:55.067867 25058 leader_election.cc:302] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:55.068167 26642 raft_consensus.cc:2781] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:29:55.068758 25259 raft_consensus.cc:2396] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5bf90e514781418da921c046aa5d9ced for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.069397 25123 leader_election.cc:288] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 4 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:55.070680 25123 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f801dcce3bfc42bf89a41156826a9203"
dest_uuid: "5bf90e514781418da921c046aa5d9ced"
from {username='slave'} at 127.0.0.1:52128
I20240502 21:29:55.070833 25123 raft_consensus.cc:482] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.070951 25123 raft_consensus.cc:3037] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:55.073758 25123 raft_consensus.cc:504] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.074941 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f801dcce3bfc42bf89a41156826a9203" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.075202 24856 raft_consensus.cc:3032] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:55.075313 24856 raft_consensus.cc:729] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:55.075681 25123 leader_election.cc:288] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 3 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:55.075723 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f801dcce3bfc42bf89a41156826a9203" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.075667 24856 consensus_queue.cc:260] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.075910 24991 raft_consensus.cc:3037] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:55.076107 24856 raft_consensus.cc:3037] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:55.078351 25123 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "87459ddf48034b28b7b3e0d0d7de91f1"
dest_uuid: "5bf90e514781418da921c046aa5d9ced"
from {username='slave'} at 127.0.0.1:52128
I20240502 21:29:55.078542 25123 raft_consensus.cc:482] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.078648 25123 raft_consensus.cc:3037] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:55.078990 24856 raft_consensus.cc:2454] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 3.
I20240502 21:29:55.079416 24991 raft_consensus.cc:2454] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 3.
I20240502 21:29:55.079438 25058 leader_election.cc:302] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:55.083389 25123 raft_consensus.cc:504] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.084718 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87459ddf48034b28b7b3e0d0d7de91f1" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.085004 25259 raft_consensus.cc:3032] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:55.085108 25259 raft_consensus.cc:729] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:55.085479 25259 consensus_queue.cc:260] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.085789 25259 raft_consensus.cc:3037] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:55.089543 25259 raft_consensus.cc:2396] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5bf90e514781418da921c046aa5d9ced for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:55.093152 25123 leader_election.cc:288] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 3 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:55.094293 25121 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3c1ce7cf99164a9a848cbb22ad2052ea"
dest_uuid: "5bf90e514781418da921c046aa5d9ced"
from {username='slave'} at 127.0.0.1:52128
I20240502 21:29:55.094491 25121 raft_consensus.cc:482] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.094588 25121 raft_consensus.cc:3037] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.097523 25121 raft_consensus.cc:504] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.098066 25121 leader_election.cc:288] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 4 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:55.099221 25123 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "083a9351f8ee4a44a86cb2d42f1ec558"
dest_uuid: "5bf90e514781418da921c046aa5d9ced"
from {username='slave'} at 127.0.0.1:52128
I20240502 21:29:55.099448 25123 raft_consensus.cc:482] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.099598 25123 raft_consensus.cc:3037] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.099720 27268 heartbeater.cc:503] Master 127.29.82.126:45283 was elected leader, sending a full tablet report...
I20240502 21:29:55.101322 27408 raft_consensus.cc:2781] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:55.104024 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87459ddf48034b28b7b3e0d0d7de91f1" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.104346 24991 raft_consensus.cc:3037] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:55.104533 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1ce7cf99164a9a848cbb22ad2052ea" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.104854 24856 raft_consensus.cc:3032] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:29:55.104983 24856 raft_consensus.cc:729] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:55.105376 24856 consensus_queue.cc:260] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.105876 24856 raft_consensus.cc:3037] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.108312 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1ce7cf99164a9a848cbb22ad2052ea" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.108595 24989 raft_consensus.cc:3037] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.108937 24991 raft_consensus.cc:2454] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 3.
I20240502 21:29:55.102994 25123 raft_consensus.cc:504] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.109798 25123 leader_election.cc:288] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 4 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:55.110698 25121 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d643c15afbec4933a0d0a145f17f703c"
dest_uuid: "5bf90e514781418da921c046aa5d9ced"
from {username='slave'} at 127.0.0.1:52128
I20240502 21:29:55.110844 25121 raft_consensus.cc:482] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.110963 25121 raft_consensus.cc:3037] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.112823 24855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "083a9351f8ee4a44a86cb2d42f1ec558" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.113116 24855 raft_consensus.cc:3037] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.113941 25121 raft_consensus.cc:504] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.114495 25121 leader_election.cc:288] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 4 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:55.115382 25124 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5360f56e93ef486b94bebcddf2edfaaa"
dest_uuid: "5bf90e514781418da921c046aa5d9ced"
from {username='slave'} at 127.0.0.1:52128
I20240502 21:29:55.115557 25124 raft_consensus.cc:482] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.115696 25124 raft_consensus.cc:3037] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.116372 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d643c15afbec4933a0d0a145f17f703c" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.116588 25260 raft_consensus.cc:3032] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:29:55.116695 25260 raft_consensus.cc:729] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:55.116979 25260 consensus_queue.cc:260] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.117301 25260 raft_consensus.cc:3037] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.118271 25124 raft_consensus.cc:504] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.103765 27408 raft_consensus.cc:686] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [term 3 LEADER]: Becoming Leader. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.118671 27408 consensus_queue.cc:237] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.118883 25124 leader_election.cc:288] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 4 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:55.119884 25124 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "52153a02d80043f0a1840b0e5aadd9b3"
dest_uuid: "5bf90e514781418da921c046aa5d9ced"
from {username='slave'} at 127.0.0.1:52128
I20240502 21:29:55.120083 25124 raft_consensus.cc:482] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.120357 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5360f56e93ef486b94bebcddf2edfaaa" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.120558 24857 raft_consensus.cc:3037] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.120801 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d643c15afbec4933a0d0a145f17f703c" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.120990 24991 raft_consensus.cc:3037] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.121060 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5360f56e93ef486b94bebcddf2edfaaa" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.121233 25261 raft_consensus.cc:3032] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:29:55.121317 25261 raft_consensus.cc:729] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:55.121580 25261 consensus_queue.cc:260] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.121870 25261 raft_consensus.cc:3037] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.124246 24991 raft_consensus.cc:2454] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 4.
I20240502 21:29:55.124681 24857 raft_consensus.cc:2396] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5bf90e514781418da921c046aa5d9ced for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:55.124852 26642 raft_consensus.cc:686] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 4 LEADER]: Becoming Leader. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.124902 25124 raft_consensus.cc:3037] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:55.125059 25261 raft_consensus.cc:2396] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5bf90e514781418da921c046aa5d9ced for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:55.125116 26642 consensus_queue.cc:237] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.126961 24855 raft_consensus.cc:2396] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5bf90e514781418da921c046aa5d9ced for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:55.127912 25124 raft_consensus.cc:504] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.128582 25124 leader_election.cc:288] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 3 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:55.129500 25123 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f62981fdfb2349bd9fbe0e85b77ae492"
dest_uuid: "5bf90e514781418da921c046aa5d9ced"
from {username='slave'} at 127.0.0.1:52128
I20240502 21:29:55.129694 25123 raft_consensus.cc:482] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.129784 25123 raft_consensus.cc:3037] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.129609 24711 catalog_manager.cc:5484] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced reported cstate change: term changed from 3 to 4, leader changed from 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68) to 5bf90e514781418da921c046aa5d9ced (127.29.82.67). New cstate: current_term: 4 leader_uuid: "5bf90e514781418da921c046aa5d9ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:55.132104 25123 raft_consensus.cc:504] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.132665 25123 leader_election.cc:288] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 4 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:55.133455 25121 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "69afb121cb564de3a9bec5e3f1cc0492"
dest_uuid: "5bf90e514781418da921c046aa5d9ced"
from {username='slave'} at 127.0.0.1:52128
I20240502 21:29:55.133651 25121 raft_consensus.cc:482] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.133742 25121 raft_consensus.cc:3037] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.136586 25121 raft_consensus.cc:504] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.137257 25121 leader_election.cc:288] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 4 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:55.137318 24711 catalog_manager.cc:5484] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced reported cstate change: term changed from 2 to 3, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 5bf90e514781418da921c046aa5d9ced (127.29.82.67). New cstate: current_term: 3 leader_uuid: "5bf90e514781418da921c046aa5d9ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:55.138100 25120 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a3a378f7a44648c3accc9dcba847040d"
dest_uuid: "5bf90e514781418da921c046aa5d9ced"
from {username='slave'} at 127.0.0.1:52128
I20240502 21:29:55.138304 25120 raft_consensus.cc:482] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.138418 25120 raft_consensus.cc:3037] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.138582 25058 leader_election.cc:302] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced; no voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b
I20240502 21:29:55.138849 27408 raft_consensus.cc:2726] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:29:55.139024 25058 leader_election.cc:302] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557; no voters: 45b7d5b9696b4046b522c880c748fe6c
I20240502 21:29:55.139279 27408 raft_consensus.cc:2781] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:55.139423 27408 raft_consensus.cc:686] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [term 3 LEADER]: Becoming Leader. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.139433 25058 leader_election.cc:302] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:55.139710 27408 consensus_queue.cc:237] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.140367 26642 raft_consensus.cc:2781] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:29:55.140487 26642 raft_consensus.cc:686] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 4 LEADER]: Becoming Leader. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.140712 26642 consensus_queue.cc:237] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.141920 25120 raft_consensus.cc:504] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.143209 25120 leader_election.cc:288] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 4 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:55.144743 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52153a02d80043f0a1840b0e5aadd9b3" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.144975 24991 raft_consensus.cc:3037] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:55.145098 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62981fdfb2349bd9fbe0e85b77ae492" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.145368 24992 raft_consensus.cc:3037] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.147773 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69afb121cb564de3a9bec5e3f1cc0492" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.148067 24990 raft_consensus.cc:3037] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.148072 24991 raft_consensus.cc:2454] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 3.
I20240502 21:29:55.152308 25258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69afb121cb564de3a9bec5e3f1cc0492" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.152572 25258 raft_consensus.cc:3032] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:29:55.152688 25258 raft_consensus.cc:729] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:55.153023 25258 consensus_queue.cc:260] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.153458 25258 raft_consensus.cc:3037] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.158385 25260 raft_consensus.cc:2396] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5bf90e514781418da921c046aa5d9ced for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.160285 25258 raft_consensus.cc:2396] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5bf90e514781418da921c046aa5d9ced for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.160583 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52153a02d80043f0a1840b0e5aadd9b3" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.160703 25261 raft_consensus.cc:3032] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:55.160768 25261 raft_consensus.cc:729] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:55.160909 25261 consensus_queue.cc:260] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.161099 25261 raft_consensus.cc:3037] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:55.143847 24854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62981fdfb2349bd9fbe0e85b77ae492" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.162313 24854 raft_consensus.cc:3032] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:29:55.162451 24854 raft_consensus.cc:729] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:55.162544 24990 raft_consensus.cc:2454] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 4.
I20240502 21:29:55.162869 24854 consensus_queue.cc:260] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.163359 24854 raft_consensus.cc:3037] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.163422 25261 raft_consensus.cc:2396] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5bf90e514781418da921c046aa5d9ced for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:55.163950 24856 raft_consensus.cc:2454] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 4.
I20240502 21:29:55.164253 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "083a9351f8ee4a44a86cb2d42f1ec558" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.164414 25259 raft_consensus.cc:3032] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:29:55.164469 25259 raft_consensus.cc:729] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:55.164671 25259 consensus_queue.cc:260] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.164840 25259 raft_consensus.cc:3037] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.166417 25259 raft_consensus.cc:2396] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5bf90e514781418da921c046aa5d9ced for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:55.166651 25123 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "175511733b714d2ea3cf877a0e5df418"
dest_uuid: "5bf90e514781418da921c046aa5d9ced"
from {username='slave'} at 127.0.0.1:52128
I20240502 21:29:55.166766 25123 raft_consensus.cc:482] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.166848 25123 raft_consensus.cc:3037] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:55.166997 24854 raft_consensus.cc:2454] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 4.
I20240502 21:29:55.167846 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a378f7a44648c3accc9dcba847040d" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.168041 24990 raft_consensus.cc:3037] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.168008 25058 leader_election.cc:302] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:55.168282 27408 raft_consensus.cc:2781] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:29:55.168437 27408 raft_consensus.cc:686] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 4 LEADER]: Becoming Leader. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.168656 27408 consensus_queue.cc:237] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.169692 25058 leader_election.cc:302] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:55.169955 27407 raft_consensus.cc:2781] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:29:55.170064 27407 raft_consensus.cc:686] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [term 4 LEADER]: Becoming Leader. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.170310 27407 consensus_queue.cc:237] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.170634 24990 raft_consensus.cc:2454] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 4.
I20240502 21:29:55.171005 25059 leader_election.cc:302] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced; no voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b
I20240502 21:29:55.171459 27408 raft_consensus.cc:2726] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:29:55.159279 24857 raft_consensus.cc:1261] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Refusing update from remote peer 45b7d5b9696b4046b522c880c748fe6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:29:55.171700 25058 leader_election.cc:302] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557; no voters: 45b7d5b9696b4046b522c880c748fe6c
I20240502 21:29:55.172116 25058 leader_election.cc:302] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557; no voters: 45b7d5b9696b4046b522c880c748fe6c
I20240502 21:29:55.172371 27408 raft_consensus.cc:2781] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:55.172513 27408 raft_consensus.cc:686] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced [term 3 LEADER]: Becoming Leader. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.172541 26870 consensus_queue.cc:1035] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:55.172762 27408 consensus_queue.cc:237] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.173159 26642 raft_consensus.cc:2781] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:29:55.173202 25123 raft_consensus.cc:504] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.173826 25123 leader_election.cc:288] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 3 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:55.174248 25058 leader_election.cc:302] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:55.174832 27407 raft_consensus.cc:2781] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:29:55.175047 27407 raft_consensus.cc:686] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced [term 4 LEADER]: Becoming Leader. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.174994 24855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175511733b714d2ea3cf877a0e5df418" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.175309 27407 consensus_queue.cc:237] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.175448 24855 raft_consensus.cc:3032] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:55.175552 24855 raft_consensus.cc:729] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:55.175706 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175511733b714d2ea3cf877a0e5df418" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.175873 24855 consensus_queue.cc:260] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.176237 24855 raft_consensus.cc:3037] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:55.176007 24990 raft_consensus.cc:3037] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:55.177280 24992 raft_consensus.cc:2454] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 4.
I20240502 21:29:55.177743 24989 raft_consensus.cc:2454] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 4.
I20240502 21:29:55.178965 24855 raft_consensus.cc:2454] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 3.
I20240502 21:29:55.179437 25058 leader_election.cc:302] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:55.179601 24990 raft_consensus.cc:2454] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 3.
I20240502 21:29:55.179862 27408 raft_consensus.cc:2781] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:55.180006 27408 raft_consensus.cc:686] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced [term 3 LEADER]: Becoming Leader. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.180039 25123 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "89c4eee02b444dc188172fd8c7b411bb"
dest_uuid: "5bf90e514781418da921c046aa5d9ced"
from {username='slave'} at 127.0.0.1:52128
I20240502 21:29:55.180061 24854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a378f7a44648c3accc9dcba847040d" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.180207 25123 raft_consensus.cc:482] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.180253 24854 raft_consensus.cc:3032] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:29:55.180312 25123 raft_consensus.cc:3037] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.180342 24854 raft_consensus.cc:729] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:55.180274 27408 consensus_queue.cc:237] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.181854 24854 consensus_queue.cc:260] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.182977 24854 raft_consensus.cc:3037] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.182987 25123 raft_consensus.cc:504] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.183637 25123 leader_election.cc:288] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 4 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:55.184016 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c4eee02b444dc188172fd8c7b411bb" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.184183 24990 raft_consensus.cc:3037] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.185218 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c4eee02b444dc188172fd8c7b411bb" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.185456 25259 raft_consensus.cc:3032] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:29:55.185567 25259 raft_consensus.cc:729] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:55.186089 25259 consensus_queue.cc:260] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.186390 25259 raft_consensus.cc:3037] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.186813 26642 raft_consensus.cc:686] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced [term 4 LEADER]: Becoming Leader. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.187068 26642 consensus_queue.cc:237] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.187999 24990 raft_consensus.cc:2454] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 4.
I20240502 21:29:55.188445 25058 leader_election.cc:302] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:55.188745 26642 raft_consensus.cc:2781] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:29:55.188872 26642 raft_consensus.cc:686] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced [term 4 LEADER]: Becoming Leader. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.189139 26642 consensus_queue.cc:237] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.190958 24854 raft_consensus.cc:2454] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 4.
I20240502 21:29:55.191051 25120 raft_consensus.cc:1261] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Refusing update from remote peer 45b7d5b9696b4046b522c880c748fe6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:29:55.191288 25123 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e3d3eab014a466fa4a421f7d9d20fba"
dest_uuid: "5bf90e514781418da921c046aa5d9ced"
from {username='slave'} at 127.0.0.1:52128
I20240502 21:29:55.191499 25123 raft_consensus.cc:482] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.191643 25123 raft_consensus.cc:3037] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.191789 27358 consensus_queue.cc:1035] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:55.193769 25259 raft_consensus.cc:2396] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5bf90e514781418da921c046aa5d9ced for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.195963 25123 raft_consensus.cc:504] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.197046 24855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3d3eab014a466fa4a421f7d9d20fba" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.197255 24855 raft_consensus.cc:3032] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:29:55.197331 24855 raft_consensus.cc:729] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:55.197577 24855 consensus_queue.cc:260] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.198009 24855 raft_consensus.cc:3037] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.201431 24855 raft_consensus.cc:2454] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 4.
I20240502 21:29:55.202039 25058 leader_election.cc:302] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:55.202044 24711 catalog_manager.cc:5484] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced reported cstate change: term changed from 3 to 4, leader changed from 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68) to 5bf90e514781418da921c046aa5d9ced (127.29.82.67). New cstate: current_term: 4 leader_uuid: "5bf90e514781418da921c046aa5d9ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:55.202399 27414 raft_consensus.cc:2781] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:29:55.202524 27414 raft_consensus.cc:686] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced [term 4 LEADER]: Becoming Leader. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.203088 27414 consensus_queue.cc:237] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.202610 24711 catalog_manager.cc:5484] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced reported cstate change: term changed from 3 to 4, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 5bf90e514781418da921c046aa5d9ced (127.29.82.67). New cstate: current_term: 4 leader_uuid: "5bf90e514781418da921c046aa5d9ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } }
I20240502 21:29:55.205201 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3d3eab014a466fa4a421f7d9d20fba" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.205430 24990 raft_consensus.cc:3037] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.205457 24711 catalog_manager.cc:5484] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced reported cstate change: term changed from 3 to 4, leader changed from 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68) to 5bf90e514781418da921c046aa5d9ced (127.29.82.67). New cstate: current_term: 4 leader_uuid: "5bf90e514781418da921c046aa5d9ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:55.205888 24711 catalog_manager.cc:5484] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced reported cstate change: term changed from 2 to 3, leader changed from 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68) to 5bf90e514781418da921c046aa5d9ced (127.29.82.67). New cstate: current_term: 3 leader_uuid: "5bf90e514781418da921c046aa5d9ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:55.206327 24711 catalog_manager.cc:5484] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced reported cstate change: term changed from 3 to 4, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 5bf90e514781418da921c046aa5d9ced (127.29.82.67). New cstate: current_term: 4 leader_uuid: "5bf90e514781418da921c046aa5d9ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:55.208636 24990 raft_consensus.cc:2454] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 4.
I20240502 21:29:55.209257 25123 leader_election.cc:288] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 4 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:55.209062 24711 catalog_manager.cc:5484] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced reported cstate change: term changed from 2 to 3, leader changed from 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68) to 5bf90e514781418da921c046aa5d9ced (127.29.82.67). New cstate: current_term: 3 leader_uuid: "5bf90e514781418da921c046aa5d9ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:55.209556 24711 catalog_manager.cc:5484] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced reported cstate change: term changed from 3 to 4, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 5bf90e514781418da921c046aa5d9ced (127.29.82.67). New cstate: current_term: 4 leader_uuid: "5bf90e514781418da921c046aa5d9ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } }
I20240502 21:29:55.210798 25123 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b4874706759a403992f9662c84a62b91"
dest_uuid: "5bf90e514781418da921c046aa5d9ced"
from {username='slave'} at 127.0.0.1:52128
I20240502 21:29:55.220736 25123 raft_consensus.cc:482] T b4874706759a403992f9662c84a62b91 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.221501 25123 raft_consensus.cc:3037] T b4874706759a403992f9662c84a62b91 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.224282 25123 raft_consensus.cc:504] T b4874706759a403992f9662c84a62b91 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.224902 24711 catalog_manager.cc:5484] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced reported cstate change: term changed from 2 to 3, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 5bf90e514781418da921c046aa5d9ced (127.29.82.67). New cstate: current_term: 3 leader_uuid: "5bf90e514781418da921c046aa5d9ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:55.225350 24855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4874706759a403992f9662c84a62b91" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.225593 24855 raft_consensus.cc:3037] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.228324 24855 raft_consensus.cc:2396] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5bf90e514781418da921c046aa5d9ced for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:55.228754 25123 leader_election.cc:288] T b4874706759a403992f9662c84a62b91 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 4 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:55.229297 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4874706759a403992f9662c84a62b91" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.229600 25259 raft_consensus.cc:3032] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:29:55.229739 25259 raft_consensus.cc:729] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:55.230036 25259 consensus_queue.cc:260] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.230355 25259 raft_consensus.cc:3037] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.228809 24711 catalog_manager.cc:5484] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced reported cstate change: term changed from 3 to 4, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 5bf90e514781418da921c046aa5d9ced (127.29.82.67). New cstate: current_term: 4 leader_uuid: "5bf90e514781418da921c046aa5d9ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:55.231192 24711 catalog_manager.cc:5484] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced reported cstate change: term changed from 3 to 4, leader changed from 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68) to 5bf90e514781418da921c046aa5d9ced (127.29.82.67). New cstate: current_term: 4 leader_uuid: "5bf90e514781418da921c046aa5d9ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:55.231563 25123 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "acf184a70440492c80cfaea7d80323de"
dest_uuid: "5bf90e514781418da921c046aa5d9ced"
from {username='slave'} at 127.0.0.1:52128
I20240502 21:29:55.231817 25123 raft_consensus.cc:482] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.231976 25123 raft_consensus.cc:3037] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:55.233124 25259 raft_consensus.cc:2396] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5bf90e514781418da921c046aa5d9ced for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:55.233561 25059 leader_election.cc:302] T b4874706759a403992f9662c84a62b91 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced; no voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b
I20240502 21:29:55.233958 27414 raft_consensus.cc:2726] T b4874706759a403992f9662c84a62b91 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:29:55.236789 25123 raft_consensus.cc:504] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.237640 25123 leader_election.cc:288] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 3 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:55.238107 24855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf184a70440492c80cfaea7d80323de" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.238327 24855 raft_consensus.cc:3037] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:55.241233 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf184a70440492c80cfaea7d80323de" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.241534 25259 raft_consensus.cc:3032] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:55.241627 25259 raft_consensus.cc:729] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:55.241935 25259 consensus_queue.cc:260] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.242111 24855 raft_consensus.cc:2454] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 3.
I20240502 21:29:55.242343 25259 raft_consensus.cc:3037] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:55.242491 25058 leader_election.cc:302] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:55.242810 27414 raft_consensus.cc:2781] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:55.242952 27414 raft_consensus.cc:686] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced [term 3 LEADER]: Becoming Leader. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.243168 27414 consensus_queue.cc:237] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.247713 25123 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c574664a57444c95b70d754cc46e313e"
dest_uuid: "5bf90e514781418da921c046aa5d9ced"
from {username='slave'} at 127.0.0.1:52128
I20240502 21:29:55.247958 25123 raft_consensus.cc:482] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.248083 25123 raft_consensus.cc:3037] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:55.251919 25123 raft_consensus.cc:504] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.252566 25259 raft_consensus.cc:2396] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5bf90e514781418da921c046aa5d9ced for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:55.254076 24855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c574664a57444c95b70d754cc46e313e" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.254350 24855 raft_consensus.cc:3032] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:55.254467 24855 raft_consensus.cc:729] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:55.254832 24855 consensus_queue.cc:260] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.255323 24855 raft_consensus.cc:3037] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:55.257476 24706 catalog_manager.cc:5484] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced reported cstate change: term changed from 2 to 3, leader changed from 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68) to 5bf90e514781418da921c046aa5d9ced (127.29.82.67). New cstate: current_term: 3 leader_uuid: "5bf90e514781418da921c046aa5d9ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } }
I20240502 21:29:55.259004 24855 raft_consensus.cc:2454] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 3.
I20240502 21:29:55.259434 25058 leader_election.cc:302] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:55.259524 25123 leader_election.cc:288] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 3 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:55.259889 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c574664a57444c95b70d754cc46e313e" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.260027 27414 raft_consensus.cc:2781] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:55.260126 24990 raft_consensus.cc:3037] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:55.260186 27414 raft_consensus.cc:686] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced [term 3 LEADER]: Becoming Leader. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.260432 27414 consensus_queue.cc:237] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.262794 24990 raft_consensus.cc:2454] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 3.
I20240502 21:29:55.263089 25123 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f25dfc6b0c04d878cd3db58d37cddec"
dest_uuid: "5bf90e514781418da921c046aa5d9ced"
from {username='slave'} at 127.0.0.1:52128
I20240502 21:29:55.263296 25123 raft_consensus.cc:482] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.263413 25123 raft_consensus.cc:3037] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.263818 24706 catalog_manager.cc:5484] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced reported cstate change: term changed from 2 to 3, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 5bf90e514781418da921c046aa5d9ced (127.29.82.67). New cstate: current_term: 3 leader_uuid: "5bf90e514781418da921c046aa5d9ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:55.266649 25123 raft_consensus.cc:504] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.267603 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f25dfc6b0c04d878cd3db58d37cddec" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.267899 24990 raft_consensus.cc:3037] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.267982 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f25dfc6b0c04d878cd3db58d37cddec" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.268486 25259 raft_consensus.cc:3032] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:29:55.268615 25259 raft_consensus.cc:729] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:55.268963 25259 consensus_queue.cc:260] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.269277 25259 raft_consensus.cc:3037] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.270814 24990 raft_consensus.cc:2454] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 4.
I20240502 21:29:55.271225 25058 leader_election.cc:302] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:55.271538 27414 raft_consensus.cc:2781] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:29:55.272075 25259 raft_consensus.cc:2396] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5bf90e514781418da921c046aa5d9ced for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.272202 27414 raft_consensus.cc:686] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [term 4 LEADER]: Becoming Leader. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.272759 25123 leader_election.cc:288] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 4 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:55.273281 27414 consensus_queue.cc:237] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.274042 25123 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "69df123d92d74693b4654c6a809bbe89"
dest_uuid: "5bf90e514781418da921c046aa5d9ced"
from {username='slave'} at 127.0.0.1:52128
I20240502 21:29:55.274255 25123 raft_consensus.cc:482] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.274384 25123 raft_consensus.cc:3037] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:55.276700 24707 catalog_manager.cc:5484] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced reported cstate change: term changed from 3 to 4, leader changed from 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68) to 5bf90e514781418da921c046aa5d9ced (127.29.82.67). New cstate: current_term: 4 leader_uuid: "5bf90e514781418da921c046aa5d9ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:55.277303 25123 raft_consensus.cc:504] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.278262 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69df123d92d74693b4654c6a809bbe89" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.278525 24990 raft_consensus.cc:2379] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5bf90e514781418da921c046aa5d9ced in current term 3: Already voted for candidate 45b7d5b9696b4046b522c880c748fe6c in this term.
I20240502 21:29:55.278975 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69df123d92d74693b4654c6a809bbe89" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.279230 25259 raft_consensus.cc:2379] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Leader election vote request: Denying vote to candidate 5bf90e514781418da921c046aa5d9ced in current term 3: Already voted for candidate 45b7d5b9696b4046b522c880c748fe6c in this term.
I20240502 21:29:55.279636 25059 leader_election.cc:302] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced; no voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557
I20240502 21:29:55.279469 25123 leader_election.cc:288] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 3 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:55.280246 27414 raft_consensus.cc:2726] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 21:29:55.281163 25123 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e145bbec053d4028ad7bd38d25dd4315"
dest_uuid: "5bf90e514781418da921c046aa5d9ced"
from {username='slave'} at 127.0.0.1:52128
I20240502 21:29:55.281339 25123 raft_consensus.cc:482] T e145bbec053d4028ad7bd38d25dd4315 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.281446 25123 raft_consensus.cc:3037] T e145bbec053d4028ad7bd38d25dd4315 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.283831 25123 raft_consensus.cc:504] T e145bbec053d4028ad7bd38d25dd4315 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.284799 25123 leader_election.cc:288] T e145bbec053d4028ad7bd38d25dd4315 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 4 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:55.284902 24855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e145bbec053d4028ad7bd38d25dd4315" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.285176 24855 raft_consensus.cc:3037] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.287228 24855 raft_consensus.cc:2396] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5bf90e514781418da921c046aa5d9ced for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:55.288412 25123 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "22e48679f46b4b72aa0f0736c76be6f1"
dest_uuid: "5bf90e514781418da921c046aa5d9ced"
from {username='slave'} at 127.0.0.1:52128
I20240502 21:29:55.288611 25123 raft_consensus.cc:482] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.288736 25123 raft_consensus.cc:3037] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:55.291216 25123 raft_consensus.cc:504] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.291879 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e145bbec053d4028ad7bd38d25dd4315" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.292196 24855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22e48679f46b4b72aa0f0736c76be6f1" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.292410 24855 raft_consensus.cc:3037] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:55.295023 24855 raft_consensus.cc:2454] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 3.
I20240502 21:29:55.295320 25259 raft_consensus.cc:3032] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:29:55.295423 25259 raft_consensus.cc:729] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:55.295498 25058 leader_election.cc:302] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:55.295817 27414 raft_consensus.cc:2781] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:55.295758 25259 consensus_queue.cc:260] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.295987 27414 raft_consensus.cc:686] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced [term 3 LEADER]: Becoming Leader. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.296056 25259 raft_consensus.cc:3037] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.296351 27414 consensus_queue.cc:237] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.298168 25259 raft_consensus.cc:2396] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5bf90e514781418da921c046aa5d9ced for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:55.298401 25123 leader_election.cc:288] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 3 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:55.299774 24707 catalog_manager.cc:5484] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced reported cstate change: term changed from 2 to 3, leader changed from 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68) to 5bf90e514781418da921c046aa5d9ced (127.29.82.67). New cstate: current_term: 3 leader_uuid: "5bf90e514781418da921c046aa5d9ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:55.302346 25123 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3e32a87a85274194916bef376c7e558e"
dest_uuid: "5bf90e514781418da921c046aa5d9ced"
from {username='slave'} at 127.0.0.1:52128
I20240502 21:29:55.302546 25123 raft_consensus.cc:482] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.302675 25123 raft_consensus.cc:3037] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.304695 25123 raft_consensus.cc:504] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.305276 25123 leader_election.cc:288] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 4 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:55.305783 24855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e32a87a85274194916bef376c7e558e" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.306372 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e32a87a85274194916bef376c7e558e" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.306571 25124 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "89b06cd6b69d41d0b49ad40f605a3e16"
dest_uuid: "5bf90e514781418da921c046aa5d9ced"
from {username='slave'} at 127.0.0.1:52128
I20240502 21:29:55.306628 24990 raft_consensus.cc:3037] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.306764 25124 raft_consensus.cc:482] T 89b06cd6b69d41d0b49ad40f605a3e16 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.306876 25124 raft_consensus.cc:3037] T 89b06cd6b69d41d0b49ad40f605a3e16 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.309655 24990 raft_consensus.cc:2454] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 4.
I20240502 21:29:55.309648 25124 raft_consensus.cc:504] T 89b06cd6b69d41d0b49ad40f605a3e16 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.310329 25059 leader_election.cc:302] T e145bbec053d4028ad7bd38d25dd4315 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced; no voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b
I20240502 21:29:55.306056 24855 raft_consensus.cc:3032] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:29:55.310617 24855 raft_consensus.cc:729] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:55.310420 25124 leader_election.cc:288] T 89b06cd6b69d41d0b49ad40f605a3e16 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 4 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:55.311089 25058 leader_election.cc:302] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:55.311122 26642 raft_consensus.cc:2726] T e145bbec053d4028ad7bd38d25dd4315 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:29:55.311532 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22e48679f46b4b72aa0f0736c76be6f1" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.311527 24855 consensus_queue.cc:260] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.311878 25259 raft_consensus.cc:3032] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:55.311975 25259 raft_consensus.cc:729] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:55.312068 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b06cd6b69d41d0b49ad40f605a3e16" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.312289 24856 raft_consensus.cc:3037] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.312460 24855 raft_consensus.cc:3037] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.313133 25259 consensus_queue.cc:260] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.313558 24854 raft_consensus.cc:1261] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Refusing update from remote peer 45b7d5b9696b4046b522c880c748fe6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 21:29:55.313591 25259 raft_consensus.cc:3037] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:55.313709 24854 pending_rounds.cc:76] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b: Aborting all ops after (but not including) 3
I20240502 21:29:55.313836 24854 pending_rounds.cc:97] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 21:29:55.314038 24854 raft_consensus.cc:2866] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:29:55.314937 27358 consensus_queue.cc:1035] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:55.315994 24855 raft_consensus.cc:2454] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 4.
I20240502 21:29:55.316129 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b06cd6b69d41d0b49ad40f605a3e16" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.316321 25261 raft_consensus.cc:3032] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:29:55.316437 25261 raft_consensus.cc:729] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:55.316677 25261 consensus_queue.cc:260] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.316975 25261 raft_consensus.cc:3037] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.317423 25259 raft_consensus.cc:2454] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 3.
I20240502 21:29:55.321203 26642 raft_consensus.cc:2781] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:29:55.321365 26642 raft_consensus.cc:686] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [term 4 LEADER]: Becoming Leader. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.321658 26642 consensus_queue.cc:237] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.323663 24992 raft_consensus.cc:1261] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Refusing update from remote peer 45b7d5b9696b4046b522c880c748fe6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 21:29:55.325057 26870 consensus_queue.cc:1035] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:55.329154 25261 raft_consensus.cc:2396] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5bf90e514781418da921c046aa5d9ced for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:55.329373 24856 raft_consensus.cc:2396] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5bf90e514781418da921c046aa5d9ced for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:55.329234 24707 catalog_manager.cc:5484] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced reported cstate change: term changed from 3 to 4, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 5bf90e514781418da921c046aa5d9ced (127.29.82.67). New cstate: current_term: 4 leader_uuid: "5bf90e514781418da921c046aa5d9ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:55.330935 25059 leader_election.cc:302] T 89b06cd6b69d41d0b49ad40f605a3e16 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced; no voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b
I20240502 21:29:55.331292 27414 raft_consensus.cc:2726] T 89b06cd6b69d41d0b49ad40f605a3e16 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:29:55.338220 26798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:55.340901 24992 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4127d80443f1499ab9a30935bbd4ca8e"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:55.341143 24992 raft_consensus.cc:482] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.341254 24992 raft_consensus.cc:3037] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.344537 24992 raft_consensus.cc:504] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.345613 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4127d80443f1499ab9a30935bbd4ca8e" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.345901 25123 raft_consensus.cc:3032] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:29:55.346035 25123 raft_consensus.cc:729] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.346383 25123 consensus_queue.cc:260] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.346750 25123 raft_consensus.cc:3037] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.349836 25123 raft_consensus.cc:2396] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 64d97dd1674d43ccadd7b4a38eeff557 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.350289 24992 leader_election.cc:288] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:55.350632 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4127d80443f1499ab9a30935bbd4ca8e" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.350869 25261 raft_consensus.cc:3037] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.351373 24990 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f801dcce3bfc42bf89a41156826a9203"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:55.351567 24990 raft_consensus.cc:482] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.351814 24990 raft_consensus.cc:3037] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.352666 25239 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:55.354954 24990 raft_consensus.cc:504] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.355191 25261 raft_consensus.cc:2396] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 64d97dd1674d43ccadd7b4a38eeff557 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.356204 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f801dcce3bfc42bf89a41156826a9203" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.356438 24856 raft_consensus.cc:3037] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.359432 24856 raft_consensus.cc:2454] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 4.
I20240502 21:29:55.360622 24926 leader_election.cc:302] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:55.360956 26535 raft_consensus.cc:2781] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:29:55.365738 24990 leader_election.cc:288] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 4 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:55.366904 24992 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "87459ddf48034b28b7b3e0d0d7de91f1"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:55.367162 24992 raft_consensus.cc:482] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.367376 24992 raft_consensus.cc:3037] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.370290 24992 raft_consensus.cc:504] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.360203 25124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f801dcce3bfc42bf89a41156826a9203" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.370957 25124 raft_consensus.cc:3032] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:29:55.371060 25124 raft_consensus.cc:729] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.371419 25124 consensus_queue.cc:260] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.371779 25124 raft_consensus.cc:3037] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.377430 25124 raft_consensus.cc:2396] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 64d97dd1674d43ccadd7b4a38eeff557 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.370663 26535 raft_consensus.cc:686] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.379030 27068 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:55.378998 26535 consensus_queue.cc:237] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.378516 24926 leader_election.cc:302] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557; no voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced
I20240502 21:29:55.381309 26535 raft_consensus.cc:2726] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240502 21:29:55.383729 24707 catalog_manager.cc:5484] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 3 to 4, leader changed from 5bf90e514781418da921c046aa5d9ced (127.29.82.67) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 4 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:55.385288 25103 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:29:55.388031 26935 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:55.388439 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87459ddf48034b28b7b3e0d0d7de91f1" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.388743 25259 raft_consensus.cc:3037] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.389062 24969 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:55.392319 25259 raft_consensus.cc:2396] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 64d97dd1674d43ccadd7b4a38eeff557 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:55.393443 25124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87459ddf48034b28b7b3e0d0d7de91f1" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.393805 24992 leader_election.cc:288] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 4 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:55.395015 24989 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bdecfa03390e405ca4925f8d1a40e0ca"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:55.395267 24989 raft_consensus.cc:482] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.395428 24989 raft_consensus.cc:3037] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.398759 24989 raft_consensus.cc:504] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.399922 24989 leader_election.cc:288] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:55.401440 24989 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3c1ce7cf99164a9a848cbb22ad2052ea"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:55.401726 24989 raft_consensus.cc:482] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.401886 24989 raft_consensus.cc:3037] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.404685 24989 raft_consensus.cc:504] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.405073 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdecfa03390e405ca4925f8d1a40e0ca" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.405403 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdecfa03390e405ca4925f8d1a40e0ca" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.405689 25259 raft_consensus.cc:3037] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.406646 24856 raft_consensus.cc:3037] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.407680 25124 raft_consensus.cc:3032] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:29:55.407814 25124 raft_consensus.cc:729] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.408195 25124 consensus_queue.cc:260] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.408514 25124 raft_consensus.cc:3037] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.410557 24856 raft_consensus.cc:2396] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 64d97dd1674d43ccadd7b4a38eeff557 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.411653 25124 raft_consensus.cc:2396] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 64d97dd1674d43ccadd7b4a38eeff557 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:55.412149 25259 raft_consensus.cc:2454] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 5.
I20240502 21:29:55.412218 24926 leader_election.cc:302] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557; no voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced
I20240502 21:29:55.412544 27446 raft_consensus.cc:2726] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:29:55.413039 24991 raft_consensus.cc:1261] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Refusing update from remote peer 45b7d5b9696b4046b522c880c748fe6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 21:29:55.413765 24989 leader_election.cc:288] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:55.414755 24991 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f62981fdfb2349bd9fbe0e85b77ae492"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:55.414958 24991 raft_consensus.cc:482] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.415060 24991 raft_consensus.cc:3037] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.415410 24927 leader_election.cc:302] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:55.417826 24991 raft_consensus.cc:504] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.418824 25120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62981fdfb2349bd9fbe0e85b77ae492" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.419083 25120 raft_consensus.cc:3032] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:29:55.419181 25120 raft_consensus.cc:729] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.419720 27446 raft_consensus.cc:2781] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:29:55.419878 27446 raft_consensus.cc:686] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.420168 27446 consensus_queue.cc:237] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.414024 26870 consensus_queue.cc:1035] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:55.419456 25120 consensus_queue.cc:260] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.422309 25120 raft_consensus.cc:3037] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.426548 25120 raft_consensus.cc:2396] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 64d97dd1674d43ccadd7b4a38eeff557 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:55.426957 27201 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:55.429108 24991 leader_election.cc:288] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:55.429286 25124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1ce7cf99164a9a848cbb22ad2052ea" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.429556 25124 raft_consensus.cc:3032] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:29:55.429632 25124 raft_consensus.cc:729] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.429927 25124 consensus_queue.cc:260] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.430100 24992 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d643c15afbec4933a0d0a145f17f703c"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:55.430219 25124 raft_consensus.cc:3037] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.430298 24992 raft_consensus.cc:482] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.430392 24992 raft_consensus.cc:3037] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.430223 24707 catalog_manager.cc:5484] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 3 to 5, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 5 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } }
I20240502 21:29:55.433017 25124 raft_consensus.cc:2396] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 64d97dd1674d43ccadd7b4a38eeff557 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:55.432941 24992 raft_consensus.cc:504] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.433992 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d643c15afbec4933a0d0a145f17f703c" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.434237 25259 raft_consensus.cc:3037] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.437837 25259 raft_consensus.cc:2396] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 64d97dd1674d43ccadd7b4a38eeff557 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.438949 24992 leader_election.cc:288] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:55.439661 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d643c15afbec4933a0d0a145f17f703c" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.439874 25123 raft_consensus.cc:3032] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:29:55.439957 25123 raft_consensus.cc:729] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.440248 24990 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57b40e72e8324b4ba6febf5eda24d333"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:55.440419 24990 raft_consensus.cc:482] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.440538 24990 raft_consensus.cc:3037] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.443500 25123 consensus_queue.cc:260] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.443907 25123 raft_consensus.cc:3037] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.444242 25120 raft_consensus.cc:1261] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Refusing update from remote peer 45b7d5b9696b4046b522c880c748fe6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 21:29:55.444839 27358 consensus_queue.cc:1035] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:55.448575 24990 raft_consensus.cc:504] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.413448 24854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1ce7cf99164a9a848cbb22ad2052ea" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.452348 24854 raft_consensus.cc:3037] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.452731 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b40e72e8324b4ba6febf5eda24d333" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.452944 24857 raft_consensus.cc:3037] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.428839 24855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62981fdfb2349bd9fbe0e85b77ae492" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.455479 24855 raft_consensus.cc:3037] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.456087 24854 raft_consensus.cc:2454] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 5.
I20240502 21:29:55.456564 24926 leader_election.cc:302] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b; no voters: 5bf90e514781418da921c046aa5d9ced
I20240502 21:29:55.459231 24855 raft_consensus.cc:2454] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 5.
I20240502 21:29:55.459986 24926 leader_election.cc:302] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b; no voters: 5bf90e514781418da921c046aa5d9ced
I20240502 21:29:55.460397 26535 raft_consensus.cc:2781] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:29:55.460743 27446 raft_consensus.cc:2781] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:29:55.464010 27446 raft_consensus.cc:686] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.464320 27446 consensus_queue.cc:237] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.456223 24857 raft_consensus.cc:2454] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 4.
I20240502 21:29:55.466288 24926 leader_election.cc:302] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:55.466616 27446 raft_consensus.cc:2781] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:29:55.466871 27446 raft_consensus.cc:686] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.467365 27446 consensus_queue.cc:237] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.470223 24837 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:55.480309 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b40e72e8324b4ba6febf5eda24d333" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.480553 25259 raft_consensus.cc:3032] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:29:55.480649 25259 raft_consensus.cc:729] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:55.480932 25259 consensus_queue.cc:260] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.481341 25259 raft_consensus.cc:3037] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.484582 25259 raft_consensus.cc:2454] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 4.
I20240502 21:29:55.485169 24990 leader_election.cc:288] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 4 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:55.486079 25123 raft_consensus.cc:2396] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 64d97dd1674d43ccadd7b4a38eeff557 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.486532 24990 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "69afb121cb564de3a9bec5e3f1cc0492"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:55.486644 24926 leader_election.cc:302] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557; no voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced
I20240502 21:29:55.486927 24990 raft_consensus.cc:482] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.487085 24990 raft_consensus.cc:3037] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.487233 27446 raft_consensus.cc:2726] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240502 21:29:55.490211 24990 raft_consensus.cc:504] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.491319 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69afb121cb564de3a9bec5e3f1cc0492" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.491746 25123 raft_consensus.cc:3032] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:29:55.491984 25123 raft_consensus.cc:729] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.492017 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69afb121cb564de3a9bec5e3f1cc0492" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.492251 25259 raft_consensus.cc:3037] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.492655 25123 consensus_queue.cc:260] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.493690 26535 raft_consensus.cc:686] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.494060 26535 consensus_queue.cc:237] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.495319 25259 raft_consensus.cc:2396] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 64d97dd1674d43ccadd7b4a38eeff557 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.495952 24990 leader_election.cc:288] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:55.496598 25123 raft_consensus.cc:3037] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.497977 24707 catalog_manager.cc:5484] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 4 to 5, leader changed from 5bf90e514781418da921c046aa5d9ced (127.29.82.67) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 5 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:55.502076 25123 raft_consensus.cc:2396] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 64d97dd1674d43ccadd7b4a38eeff557 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.502519 24991 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a3a378f7a44648c3accc9dcba847040d"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:55.502730 24991 raft_consensus.cc:482] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.502868 24991 raft_consensus.cc:3037] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.505699 24991 raft_consensus.cc:504] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.506388 24991 leader_election.cc:288] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:55.508116 24707 catalog_manager.cc:5484] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 4 to 5, leader changed from 5bf90e514781418da921c046aa5d9ced (127.29.82.67) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 5 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } }
I20240502 21:29:55.508651 24926 leader_election.cc:302] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557; no voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced
I20240502 21:29:55.508862 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a378f7a44648c3accc9dcba847040d" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.509109 24857 raft_consensus.cc:3037] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.509145 24707 catalog_manager.cc:5484] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 3 to 4, leader changed from 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 4 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:55.510280 24991 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "89c4eee02b444dc188172fd8c7b411bb"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:55.510496 24991 raft_consensus.cc:482] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.510603 24991 raft_consensus.cc:3037] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.512495 24857 raft_consensus.cc:2454] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 5.
I20240502 21:29:55.512714 26535 raft_consensus.cc:2726] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240502 21:29:55.513204 24926 leader_election.cc:302] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:55.513610 26535 raft_consensus.cc:2781] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:29:55.513872 26535 raft_consensus.cc:686] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.514256 26535 consensus_queue.cc:237] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.517071 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a378f7a44648c3accc9dcba847040d" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.518118 25123 raft_consensus.cc:3032] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:29:55.518256 25123 raft_consensus.cc:729] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.518096 24706 catalog_manager.cc:5484] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 4 to 5, leader changed from 5bf90e514781418da921c046aa5d9ced (127.29.82.67) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 5 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } }
I20240502 21:29:55.519722 25123 consensus_queue.cc:260] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.520061 25123 raft_consensus.cc:3037] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.522823 24991 raft_consensus.cc:504] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.524008 25124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c4eee02b444dc188172fd8c7b411bb" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.524252 25124 raft_consensus.cc:3032] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:29:55.524328 25124 raft_consensus.cc:729] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.524582 25124 consensus_queue.cc:260] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.524720 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c4eee02b444dc188172fd8c7b411bb" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.524891 25124 raft_consensus.cc:3037] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.525072 25259 raft_consensus.cc:3037] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.526047 24991 leader_election.cc:288] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:55.527113 24990 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e3d3eab014a466fa4a421f7d9d20fba"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:55.527281 24990 raft_consensus.cc:482] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.527387 24990 raft_consensus.cc:3037] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.528216 25124 raft_consensus.cc:2396] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 64d97dd1674d43ccadd7b4a38eeff557 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.530244 24990 raft_consensus.cc:504] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.530915 24990 leader_election.cc:288] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:55.531143 25259 raft_consensus.cc:2396] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 64d97dd1674d43ccadd7b4a38eeff557 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.531920 25124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3d3eab014a466fa4a421f7d9d20fba" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.532119 25124 raft_consensus.cc:3032] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:29:55.532202 25124 raft_consensus.cc:729] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.532234 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3d3eab014a466fa4a421f7d9d20fba" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.532135 24990 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "175511733b714d2ea3cf877a0e5df418"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:55.532459 24857 raft_consensus.cc:3037] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.532509 24990 raft_consensus.cc:482] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.532478 25124 consensus_queue.cc:260] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.532631 24990 raft_consensus.cc:3037] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.532799 25124 raft_consensus.cc:3037] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.533221 25123 raft_consensus.cc:2396] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 64d97dd1674d43ccadd7b4a38eeff557 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:55.532302 24927 leader_election.cc:302] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557; no voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced
I20240502 21:29:55.535302 27451 raft_consensus.cc:2726] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240502 21:29:55.535599 24857 raft_consensus.cc:2454] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 5.
I20240502 21:29:55.535773 25124 raft_consensus.cc:2396] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 64d97dd1674d43ccadd7b4a38eeff557 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:55.536103 24926 leader_election.cc:302] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:55.536428 27451 raft_consensus.cc:2781] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:29:55.536602 27451 raft_consensus.cc:686] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.536888 27451 consensus_queue.cc:237] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.539834 24990 raft_consensus.cc:504] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.540627 24990 leader_election.cc:288] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 4 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:55.543937 25124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175511733b714d2ea3cf877a0e5df418" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.544131 25124 raft_consensus.cc:3032] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:29:55.544210 25124 raft_consensus.cc:729] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.544394 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175511733b714d2ea3cf877a0e5df418" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.544617 24857 raft_consensus.cc:3037] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.544559 25124 consensus_queue.cc:260] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.544814 25124 raft_consensus.cc:3037] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.546663 25124 raft_consensus.cc:2396] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 64d97dd1674d43ccadd7b4a38eeff557 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.548314 24707 catalog_manager.cc:5484] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 4 to 5, leader changed from 5bf90e514781418da921c046aa5d9ced (127.29.82.67) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 5 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:55.548550 24857 raft_consensus.cc:2454] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 4.
I20240502 21:29:55.548946 24926 leader_election.cc:302] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b; no voters: 5bf90e514781418da921c046aa5d9ced
I20240502 21:29:55.549253 27451 raft_consensus.cc:2781] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:29:55.549414 27451 raft_consensus.cc:686] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.549714 27451 consensus_queue.cc:237] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.555485 24990 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a851d0bede1444c8a96ea45ea948bd59"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:55.555253 24707 catalog_manager.cc:5484] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 3 to 4, leader changed from 5bf90e514781418da921c046aa5d9ced (127.29.82.67) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 4 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:55.555681 24990 raft_consensus.cc:482] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.555794 24990 raft_consensus.cc:3037] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.567268 24990 raft_consensus.cc:504] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.568094 24990 leader_election.cc:288] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:55.568548 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851d0bede1444c8a96ea45ea948bd59" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.568763 24857 raft_consensus.cc:3037] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.572305 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851d0bede1444c8a96ea45ea948bd59" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.572535 25259 raft_consensus.cc:3037] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.572499 24990 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3e32a87a85274194916bef376c7e558e"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:55.572693 24990 raft_consensus.cc:482] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.572811 24990 raft_consensus.cc:3037] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.575778 24990 raft_consensus.cc:504] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.576283 25259 raft_consensus.cc:2454] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 5.
I20240502 21:29:55.576400 24990 leader_election.cc:288] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:55.576745 24927 leader_election.cc:302] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:55.577430 24855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e32a87a85274194916bef376c7e558e" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.577651 24855 raft_consensus.cc:3037] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.580042 24855 raft_consensus.cc:2454] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 5.
I20240502 21:29:55.580411 26535 raft_consensus.cc:2781] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:29:55.580459 24926 leader_election.cc:302] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:55.580591 26535 raft_consensus.cc:686] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.580816 27446 raft_consensus.cc:2781] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:29:55.580868 26535 consensus_queue.cc:237] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.581465 24990 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c574664a57444c95b70d754cc46e313e"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:55.581640 24990 raft_consensus.cc:482] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.580981 27446 raft_consensus.cc:686] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.582099 27446 consensus_queue.cc:237] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.583096 24857 raft_consensus.cc:2396] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 64d97dd1674d43ccadd7b4a38eeff557 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.581786 24990 raft_consensus.cc:3037] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.585543 25124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e32a87a85274194916bef376c7e558e" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.585827 25124 raft_consensus.cc:3032] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:29:55.585929 25124 raft_consensus.cc:729] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.586032 24707 catalog_manager.cc:5484] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 3 to 5, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 5 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:55.586344 25124 consensus_queue.cc:260] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.586632 25259 raft_consensus.cc:1261] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Refusing update from remote peer 5bf90e514781418da921c046aa5d9ced: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 21:29:55.586786 25259 pending_rounds.cc:76] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c: Aborting all ops after (but not including) 3
I20240502 21:29:55.586879 25259 pending_rounds.cc:97] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 21:29:55.587050 25259 raft_consensus.cc:2866] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:29:55.590675 24990 raft_consensus.cc:504] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.596346 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c574664a57444c95b70d754cc46e313e" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.596573 24857 raft_consensus.cc:3037] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.586660 25124 raft_consensus.cc:3037] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.599402 24857 raft_consensus.cc:2454] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 4.
I20240502 21:29:55.588663 26642 consensus_queue.cc:1035] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.011s
I20240502 21:29:55.601862 24926 leader_election.cc:302] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:55.602210 26535 raft_consensus.cc:2781] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:29:55.602329 26535 raft_consensus.cc:686] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.602599 26535 consensus_queue.cc:237] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.602953 24707 catalog_manager.cc:5484] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 4 to 5, leader changed from 5bf90e514781418da921c046aa5d9ced (127.29.82.67) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 5 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:55.604081 25124 raft_consensus.cc:2396] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 64d97dd1674d43ccadd7b4a38eeff557 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:55.607982 24990 leader_election.cc:288] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 4 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:55.609256 24991 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f25dfc6b0c04d878cd3db58d37cddec"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:55.609483 24991 raft_consensus.cc:482] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.609598 24991 raft_consensus.cc:3037] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.609869 24706 catalog_manager.cc:5484] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 3 to 4, leader changed from 5bf90e514781418da921c046aa5d9ced (127.29.82.67) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 4 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:55.612810 24991 raft_consensus.cc:504] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.613854 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f25dfc6b0c04d878cd3db58d37cddec" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.614099 25261 raft_consensus.cc:3037] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.615433 24990 raft_consensus.cc:1261] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Refusing update from remote peer 5bf90e514781418da921c046aa5d9ced: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 21:29:55.616879 25261 raft_consensus.cc:2396] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 64d97dd1674d43ccadd7b4a38eeff557 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.618052 27408 consensus_queue.cc:1035] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:55.618922 24991 leader_election.cc:288] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:55.619525 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f25dfc6b0c04d878cd3db58d37cddec" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.619796 25123 raft_consensus.cc:3032] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:29:55.619877 25123 raft_consensus.cc:729] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.620154 25123 consensus_queue.cc:260] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.620453 25123 raft_consensus.cc:3037] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.623499 25123 raft_consensus.cc:2396] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 64d97dd1674d43ccadd7b4a38eeff557 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.624688 24926 leader_election.cc:302] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64d97dd1674d43ccadd7b4a38eeff557; no voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced
I20240502 21:29:55.624975 24991 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "69df123d92d74693b4654c6a809bbe89"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:55.625159 24991 raft_consensus.cc:482] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.625254 24991 raft_consensus.cc:3037] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.626003 27451 raft_consensus.cc:2726] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240502 21:29:55.627992 24857 raft_consensus.cc:1261] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Refusing update from remote peer 5bf90e514781418da921c046aa5d9ced: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 21:29:55.628667 27408 consensus_queue.cc:1035] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:55.630954 24991 raft_consensus.cc:504] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.633687 25124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c574664a57444c95b70d754cc46e313e" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.634557 25124 raft_consensus.cc:3032] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:29:55.634690 25124 raft_consensus.cc:729] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.635090 25124 consensus_queue.cc:260] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.635905 25124 raft_consensus.cc:3037] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.636788 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69df123d92d74693b4654c6a809bbe89" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.637138 24991 leader_election.cc:288] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 4 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:55.637158 25259 raft_consensus.cc:3032] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:29:55.637274 25259 raft_consensus.cc:729] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:55.637777 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69df123d92d74693b4654c6a809bbe89" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.637825 25259 consensus_queue.cc:260] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.638267 25123 raft_consensus.cc:3037] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.638911 25259 raft_consensus.cc:3037] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.641439 25123 raft_consensus.cc:2454] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 4.
I20240502 21:29:55.641888 24926 leader_election.cc:302] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:55.643469 27451 raft_consensus.cc:2781] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:29:55.643627 27451 raft_consensus.cc:686] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.643864 27451 consensus_queue.cc:237] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.645884 24991 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fed2d260d00d470e95e9a62cc71ba94f"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:55.646090 24991 raft_consensus.cc:482] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.646232 24991 raft_consensus.cc:3037] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.650137 25259 raft_consensus.cc:2454] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 4.
I20240502 21:29:55.653537 24991 raft_consensus.cc:504] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.654268 25124 raft_consensus.cc:2396] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 64d97dd1674d43ccadd7b4a38eeff557 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.654875 24707 catalog_manager.cc:5484] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 3 to 4, leader changed from 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 4 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:55.655689 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fed2d260d00d470e95e9a62cc71ba94f" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.655911 24857 raft_consensus.cc:3037] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.658591 24857 raft_consensus.cc:2396] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 64d97dd1674d43ccadd7b4a38eeff557 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.659648 24991 leader_election.cc:288] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:55.660539 24990 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "52153a02d80043f0a1840b0e5aadd9b3"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:55.660760 24990 raft_consensus.cc:482] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.660938 24990 raft_consensus.cc:3037] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.662734 24990 raft_consensus.cc:504] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.663285 24990 leader_election.cc:288] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 4 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:55.663767 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52153a02d80043f0a1840b0e5aadd9b3" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.663677 25124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52153a02d80043f0a1840b0e5aadd9b3" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.664038 25261 raft_consensus.cc:3037] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.664057 25124 raft_consensus.cc:3032] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:29:55.664150 25124 raft_consensus.cc:729] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.664464 25124 consensus_queue.cc:260] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.664757 25124 raft_consensus.cc:3037] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.666527 25124 raft_consensus.cc:2454] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 4.
I20240502 21:29:55.666811 24991 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "986c305824414299ac30f1e50f514c8a"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:55.667016 24991 raft_consensus.cc:482] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.667034 24926 leader_election.cc:302] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:55.667394 26535 raft_consensus.cc:2781] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:29:55.667359 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fed2d260d00d470e95e9a62cc71ba94f" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.667544 26535 raft_consensus.cc:686] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.667599 25259 raft_consensus.cc:3037] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.667162 24991 raft_consensus.cc:3037] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.667809 26535 consensus_queue.cc:237] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.667958 25261 raft_consensus.cc:2454] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 4.
I20240502 21:29:55.670394 24991 raft_consensus.cc:504] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.671319 24707 catalog_manager.cc:5484] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 3 to 4, leader changed from 5bf90e514781418da921c046aa5d9ced (127.29.82.67) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 4 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:55.671495 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c305824414299ac30f1e50f514c8a" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.671655 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c305824414299ac30f1e50f514c8a" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.671787 25261 raft_consensus.cc:3037] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.671866 24857 raft_consensus.cc:3037] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.674242 24857 raft_consensus.cc:2396] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 64d97dd1674d43ccadd7b4a38eeff557 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.675642 24991 leader_election.cc:288] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:55.676559 24991 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8423ff21d93741cdab3093ea454bb532"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:55.676779 24991 raft_consensus.cc:482] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.676961 24991 raft_consensus.cc:3037] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.677016 25261 raft_consensus.cc:2454] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 5.
I20240502 21:29:55.677556 24927 leader_election.cc:302] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters: a171ad73f86643afb1444248c0d0bc8b
I20240502 21:29:55.677908 25259 raft_consensus.cc:2454] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 5.
I20240502 21:29:55.678112 27451 raft_consensus.cc:2781] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:29:55.678337 24927 leader_election.cc:302] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters: a171ad73f86643afb1444248c0d0bc8b
I20240502 21:29:55.678351 27451 raft_consensus.cc:686] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.678798 26535 raft_consensus.cc:2781] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:29:55.678952 26535 raft_consensus.cc:686] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.678964 27451 consensus_queue.cc:237] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.679234 26535 consensus_queue.cc:237] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.680011 24991 raft_consensus.cc:504] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.680734 24991 leader_election.cc:288] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:55.681107 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8423ff21d93741cdab3093ea454bb532" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.681334 24857 raft_consensus.cc:3037] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.681489 25258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8423ff21d93741cdab3093ea454bb532" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.681672 25258 raft_consensus.cc:3037] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.684278 24857 raft_consensus.cc:2396] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 64d97dd1674d43ccadd7b4a38eeff557 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.684360 25258 raft_consensus.cc:2454] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 5.
I20240502 21:29:55.684837 24927 leader_election.cc:302] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:55.685140 27446 raft_consensus.cc:2781] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:29:55.685313 27446 raft_consensus.cc:686] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.685590 27446 consensus_queue.cc:237] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.690711 24991 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "48e7c6534b544336ae216d3200f4f01f"
dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
from {username='slave'} at 127.0.0.1:33894
I20240502 21:29:55.690996 24991 raft_consensus.cc:482] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.691143 24991 raft_consensus.cc:3037] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.697016 24991 raft_consensus.cc:504] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.697103 24707 catalog_manager.cc:5484] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 3 to 5, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 5 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:55.697862 24707 catalog_manager.cc:5484] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 3 to 5, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 5 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:55.698602 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e7c6534b544336ae216d3200f4f01f" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.698911 25259 raft_consensus.cc:3037] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.703529 25259 raft_consensus.cc:2454] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d97dd1674d43ccadd7b4a38eeff557 in term 5.
I20240502 21:29:55.704772 24927 leader_election.cc:302] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:55.705168 26535 raft_consensus.cc:2781] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:29:55.705363 26535 raft_consensus.cc:686] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Becoming Leader. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.705709 26535 consensus_queue.cc:237] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.708118 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e7c6534b544336ae216d3200f4f01f" candidate_uuid: "64d97dd1674d43ccadd7b4a38eeff557" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.708355 24857 raft_consensus.cc:3037] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.707967 24991 leader_election.cc:288] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [CANDIDATE]: Term 5 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:55.712162 24857 raft_consensus.cc:2396] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 64d97dd1674d43ccadd7b4a38eeff557 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.715029 25259 raft_consensus.cc:1261] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Refusing update from remote peer 5bf90e514781418da921c046aa5d9ced: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 21:29:55.715193 25259 pending_rounds.cc:76] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c: Aborting all ops after (but not including) 3
I20240502 21:29:55.715257 25259 pending_rounds.cc:97] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 21:29:55.715338 25259 raft_consensus.cc:2866] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:29:55.716140 26642 consensus_queue.cc:1035] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:55.722438 24707 catalog_manager.cc:5484] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 3 to 5, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 5 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:55.723325 24857 raft_consensus.cc:1261] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Refusing update from remote peer 5bf90e514781418da921c046aa5d9ced: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 21:29:55.724015 26642 consensus_queue.cc:1035] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:55.724483 24855 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f801dcce3bfc42bf89a41156826a9203"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:55.724681 24855 raft_consensus.cc:482] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.724793 24855 raft_consensus.cc:3037] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.727612 24855 raft_consensus.cc:504] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.730325 25259 raft_consensus.cc:1261] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Refusing update from remote peer 5bf90e514781418da921c046aa5d9ced: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 21:29:55.730146 24711 catalog_manager.cc:5484] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 reported cstate change: term changed from 3 to 5, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66). New cstate: current_term: 5 leader_uuid: "64d97dd1674d43ccadd7b4a38eeff557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } }
I20240502 21:29:55.732033 27408 consensus_queue.cc:1035] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:55.734040 24855 leader_election.cc:288] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 5 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:55.734617 25124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f801dcce3bfc42bf89a41156826a9203" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.734826 25124 raft_consensus.cc:3037] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.737025 25124 raft_consensus.cc:2396] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.737916 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f801dcce3bfc42bf89a41156826a9203" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.738133 24991 raft_consensus.cc:3032] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:29:55.738236 24991 raft_consensus.cc:729] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.738488 24991 consensus_queue.cc:260] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.738780 24991 raft_consensus.cc:3037] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.739678 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3c1ce7cf99164a9a848cbb22ad2052ea"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:55.739846 24857 raft_consensus.cc:482] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.739966 24857 raft_consensus.cc:3037] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.741734 24991 raft_consensus.cc:2396] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.742236 24789 leader_election.cc:302] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a171ad73f86643afb1444248c0d0bc8b; no voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557
I20240502 21:29:55.742718 27306 raft_consensus.cc:2726] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240502 21:29:55.742671 24857 raft_consensus.cc:504] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.743388 24857 leader_election.cc:288] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 6 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:55.744813 25124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1ce7cf99164a9a848cbb22ad2052ea" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.745014 25124 raft_consensus.cc:3037] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.746873 25124 raft_consensus.cc:2396] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:55.747483 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "083a9351f8ee4a44a86cb2d42f1ec558"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:55.747676 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1ce7cf99164a9a848cbb22ad2052ea" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.747872 24991 raft_consensus.cc:3032] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:29:55.747768 24857 raft_consensus.cc:482] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.747977 24991 raft_consensus.cc:729] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.748131 24857 raft_consensus.cc:3037] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.748291 24991 consensus_queue.cc:260] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.748673 24991 raft_consensus.cc:3037] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.751205 24857 raft_consensus.cc:504] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.751482 24991 raft_consensus.cc:2396] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:55.751973 24789 leader_election.cc:302] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a171ad73f86643afb1444248c0d0bc8b; no voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557
I20240502 21:29:55.752259 24857 leader_election.cc:288] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 5 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:55.752779 27306 raft_consensus.cc:2726] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240502 21:29:55.753228 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "083a9351f8ee4a44a86cb2d42f1ec558" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.753445 25259 raft_consensus.cc:3037] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.753571 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f62981fdfb2349bd9fbe0e85b77ae492"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:55.753795 24857 raft_consensus.cc:482] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.753978 24857 raft_consensus.cc:3037] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.755484 25124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "083a9351f8ee4a44a86cb2d42f1ec558" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.755722 25124 raft_consensus.cc:3037] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.757185 24857 raft_consensus.cc:504] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.757375 25259 raft_consensus.cc:2396] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.758472 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62981fdfb2349bd9fbe0e85b77ae492" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.758648 25123 raft_consensus.cc:3037] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.761353 25123 raft_consensus.cc:2396] T f62981fdfb2349bd9fbe0e85b77ae492 P 5bf90e514781418da921c046aa5d9ced [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:55.762097 25124 raft_consensus.cc:2454] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 5.
I20240502 21:29:55.763887 24789 leader_election.cc:302] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b; no voters: 45b7d5b9696b4046b522c880c748fe6c
I20240502 21:29:55.764186 27306 raft_consensus.cc:2781] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:29:55.765028 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62981fdfb2349bd9fbe0e85b77ae492" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.765277 24991 raft_consensus.cc:3032] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:29:55.765395 24991 raft_consensus.cc:729] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.765738 24991 consensus_queue.cc:260] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.766105 24991 raft_consensus.cc:3037] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.769843 24991 raft_consensus.cc:2396] T f62981fdfb2349bd9fbe0e85b77ae492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:55.770430 24789 leader_election.cc:302] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a171ad73f86643afb1444248c0d0bc8b; no voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557
I20240502 21:29:55.775004 24857 leader_election.cc:288] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 6 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:55.776294 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bdecfa03390e405ca4925f8d1a40e0ca"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:55.776646 24857 raft_consensus.cc:482] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.776868 24857 raft_consensus.cc:3037] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.777071 27472 raft_consensus.cc:2726] T f62981fdfb2349bd9fbe0e85b77ae492 P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240502 21:29:55.779114 24857 raft_consensus.cc:504] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.779671 24857 leader_election.cc:288] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 6 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:55.780198 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdecfa03390e405ca4925f8d1a40e0ca" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.780457 25259 raft_consensus.cc:3037] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.781630 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdecfa03390e405ca4925f8d1a40e0ca" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.781877 24991 raft_consensus.cc:3032] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:29:55.782014 24991 raft_consensus.cc:729] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.782335 24991 consensus_queue.cc:260] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.782717 24991 raft_consensus.cc:3037] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.783463 25259 raft_consensus.cc:2454] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 6.
I20240502 21:29:55.784018 24790 leader_election.cc:302] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:55.784209 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5360f56e93ef486b94bebcddf2edfaaa"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:55.784372 24857 raft_consensus.cc:482] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.784513 24857 raft_consensus.cc:3037] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.784518 27472 raft_consensus.cc:2781] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:29:55.784817 27472 raft_consensus.cc:686] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [term 6 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:55.785171 27472 consensus_queue.cc:237] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.787246 24857 raft_consensus.cc:504] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.788142 27306 raft_consensus.cc:686] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [term 5 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:55.788411 27306 consensus_queue.cc:237] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.788791 24991 raft_consensus.cc:2396] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:55.792156 24857 leader_election.cc:288] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 5 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:55.792702 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5360f56e93ef486b94bebcddf2edfaaa" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.792979 25259 raft_consensus.cc:3037] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.793022 25124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5360f56e93ef486b94bebcddf2edfaaa" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.793677 25124 raft_consensus.cc:3037] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.796211 25259 raft_consensus.cc:2396] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.797188 25124 raft_consensus.cc:2454] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 5.
I20240502 21:29:55.797636 24789 leader_election.cc:302] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b; no voters: 45b7d5b9696b4046b522c880c748fe6c
I20240502 21:29:55.798905 27306 raft_consensus.cc:2781] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:29:55.799078 27306 raft_consensus.cc:686] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [term 5 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:55.799211 24706 catalog_manager.cc:5484] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 3 to 5, leader changed from 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 5 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } }
I20240502 21:29:55.799358 27306 consensus_queue.cc:237] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.801343 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57b40e72e8324b4ba6febf5eda24d333"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:55.801568 24857 raft_consensus.cc:482] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.801685 24857 raft_consensus.cc:3037] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.804576 24857 raft_consensus.cc:504] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.806442 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b40e72e8324b4ba6febf5eda24d333" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.806713 24990 raft_consensus.cc:3032] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:29:55.806807 24990 raft_consensus.cc:729] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.807166 24990 consensus_queue.cc:260] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.807538 24990 raft_consensus.cc:3037] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.812770 24990 raft_consensus.cc:2396] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:55.813271 24857 leader_election.cc:288] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 5 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:55.814685 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "22e48679f46b4b72aa0f0736c76be6f1"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:55.814898 24857 raft_consensus.cc:482] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.815274 24857 raft_consensus.cc:3037] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.815196 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b40e72e8324b4ba6febf5eda24d333" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.816181 25259 raft_consensus.cc:3037] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.817220 24706 catalog_manager.cc:5484] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 5 to 6, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 6 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } }
I20240502 21:29:55.818224 24857 raft_consensus.cc:504] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.819077 24857 leader_election.cc:288] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 4 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:55.819486 25258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22e48679f46b4b72aa0f0736c76be6f1" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.819792 25258 raft_consensus.cc:3037] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.820144 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3e32a87a85274194916bef376c7e558e"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:55.820333 24857 raft_consensus.cc:482] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.820441 24857 raft_consensus.cc:3037] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.822953 24857 raft_consensus.cc:504] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.823411 25258 raft_consensus.cc:2454] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 4.
I20240502 21:29:55.823549 24857 leader_election.cc:288] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 6 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:55.823886 24706 catalog_manager.cc:5484] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 3 to 5, leader changed from 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 5 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:55.825898 25124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22e48679f46b4b72aa0f0736c76be6f1" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.826020 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e32a87a85274194916bef376c7e558e" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.826155 25124 raft_consensus.cc:3032] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:29:55.826237 24990 raft_consensus.cc:3032] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:29:55.826260 25124 raft_consensus.cc:729] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.826332 24990 raft_consensus.cc:729] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.826560 25124 consensus_queue.cc:260] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.826601 24990 consensus_queue.cc:260] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.826673 25259 raft_consensus.cc:2454] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 5.
I20240502 21:29:55.826865 24990 raft_consensus.cc:3037] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.826951 25124 raft_consensus.cc:3037] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.827693 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e32a87a85274194916bef376c7e558e" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.827920 25123 raft_consensus.cc:3037] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.829891 25124 raft_consensus.cc:2454] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 4.
I20240502 21:29:55.830360 24789 leader_election.cc:302] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:55.830747 27306 raft_consensus.cc:2781] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:29:55.830984 24790 leader_election.cc:302] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters: 64d97dd1674d43ccadd7b4a38eeff557
I20240502 21:29:55.831063 27306 raft_consensus.cc:686] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [term 4 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:55.831455 27306 consensus_queue.cc:237] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.832137 25123 raft_consensus.cc:2396] T 3e32a87a85274194916bef376c7e558e P 5bf90e514781418da921c046aa5d9ced [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:55.832556 24990 raft_consensus.cc:2396] T 3e32a87a85274194916bef376c7e558e P 64d97dd1674d43ccadd7b4a38eeff557 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:55.832559 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a3a378f7a44648c3accc9dcba847040d"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:55.832787 24857 raft_consensus.cc:482] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.832952 24789 leader_election.cc:302] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a171ad73f86643afb1444248c0d0bc8b; no voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557
I20240502 21:29:55.833071 27472 raft_consensus.cc:2781] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:29:55.833187 27472 raft_consensus.cc:686] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [term 5 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:55.833330 27471 raft_consensus.cc:2726] T 3e32a87a85274194916bef376c7e558e P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240502 21:29:55.833472 27472 consensus_queue.cc:237] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.832953 24857 raft_consensus.cc:3037] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.836663 24857 raft_consensus.cc:504] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.837277 24857 leader_election.cc:288] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 6 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:55.837257 24706 catalog_manager.cc:5484] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 3 to 4, leader changed from 5bf90e514781418da921c046aa5d9ced (127.29.82.67) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 4 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:55.838248 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a378f7a44648c3accc9dcba847040d" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.838452 25123 raft_consensus.cc:3037] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.838692 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a378f7a44648c3accc9dcba847040d" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.838896 24990 raft_consensus.cc:3032] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:29:55.838989 24990 raft_consensus.cc:729] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.839215 24990 consensus_queue.cc:260] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.839411 24990 raft_consensus.cc:3037] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.840067 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a851d0bede1444c8a96ea45ea948bd59"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:55.840279 24857 raft_consensus.cc:482] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.840366 24857 raft_consensus.cc:3037] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.841125 25123 raft_consensus.cc:2396] T a3a378f7a44648c3accc9dcba847040d P 5bf90e514781418da921c046aa5d9ced [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:55.842182 24990 raft_consensus.cc:2396] T a3a378f7a44648c3accc9dcba847040d P 64d97dd1674d43ccadd7b4a38eeff557 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:55.842617 24857 raft_consensus.cc:504] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.843600 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851d0bede1444c8a96ea45ea948bd59" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.843812 25259 raft_consensus.cc:3037] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.845783 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851d0bede1444c8a96ea45ea948bd59" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.845949 25259 raft_consensus.cc:2454] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 6.
I20240502 21:29:55.845978 24990 raft_consensus.cc:3032] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:29:55.846073 24990 raft_consensus.cc:729] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.846331 24857 leader_election.cc:288] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 6 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:55.846422 24790 leader_election.cc:302] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:55.846604 24990 consensus_queue.cc:260] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.846531 24706 catalog_manager.cc:5484] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 4 to 5, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 5 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:55.846763 27483 raft_consensus.cc:2781] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:29:55.846920 27483 raft_consensus.cc:686] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [term 6 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:55.846431 24789 leader_election.cc:302] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a171ad73f86643afb1444248c0d0bc8b; no voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557
I20240502 21:29:55.847298 27306 raft_consensus.cc:2726] T a3a378f7a44648c3accc9dcba847040d P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240502 21:29:55.847250 27483 consensus_queue.cc:237] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.847735 24990 raft_consensus.cc:3037] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.847736 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "175511733b714d2ea3cf877a0e5df418"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:55.847945 24857 raft_consensus.cc:482] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.848090 24857 raft_consensus.cc:3037] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.851076 24857 raft_consensus.cc:504] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.851282 24990 raft_consensus.cc:2396] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:55.851755 24857 leader_election.cc:288] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 5 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:55.852284 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175511733b714d2ea3cf877a0e5df418" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.852505 25123 raft_consensus.cc:3037] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.852396 24706 catalog_manager.cc:5484] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 5 to 6, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 6 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:55.853111 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e3d3eab014a466fa4a421f7d9d20fba"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:55.853410 24857 raft_consensus.cc:482] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.853575 24857 raft_consensus.cc:3037] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.855926 25123 raft_consensus.cc:2396] T 175511733b714d2ea3cf877a0e5df418 P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.856245 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175511733b714d2ea3cf877a0e5df418" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.856312 24857 raft_consensus.cc:504] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.856505 24990 raft_consensus.cc:3032] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:29:55.856606 24990 raft_consensus.cc:729] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.856817 24990 consensus_queue.cc:260] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.857019 24990 raft_consensus.cc:3037] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.857183 24857 leader_election.cc:288] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 6 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:55.857960 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3d3eab014a466fa4a421f7d9d20fba" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.857968 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3d3eab014a466fa4a421f7d9d20fba" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.858144 25123 raft_consensus.cc:3037] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.858145 24991 raft_consensus.cc:3032] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:29:55.858248 24991 raft_consensus.cc:729] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.858503 24991 consensus_queue.cc:260] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.858808 24991 raft_consensus.cc:3037] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.859935 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b4874706759a403992f9662c84a62b91"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:55.860148 24857 raft_consensus.cc:482] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.860306 24857 raft_consensus.cc:3037] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.860855 25123 raft_consensus.cc:2396] T 4e3d3eab014a466fa4a421f7d9d20fba P 5bf90e514781418da921c046aa5d9ced [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:55.861658 24991 raft_consensus.cc:2396] T 4e3d3eab014a466fa4a421f7d9d20fba P 64d97dd1674d43ccadd7b4a38eeff557 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:55.862185 24789 leader_election.cc:302] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a171ad73f86643afb1444248c0d0bc8b; no voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557
I20240502 21:29:55.862602 27483 raft_consensus.cc:2726] T 4e3d3eab014a466fa4a421f7d9d20fba P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240502 21:29:55.863190 24857 raft_consensus.cc:504] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.863462 24990 raft_consensus.cc:2396] T 175511733b714d2ea3cf877a0e5df418 P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.863907 24789 leader_election.cc:302] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a171ad73f86643afb1444248c0d0bc8b; no voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557
I20240502 21:29:55.864164 27483 raft_consensus.cc:2726] T 175511733b714d2ea3cf877a0e5df418 P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240502 21:29:55.864523 24857 leader_election.cc:288] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 5 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:55.864728 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4874706759a403992f9662c84a62b91" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.864820 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4874706759a403992f9662c84a62b91" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.864909 25123 raft_consensus.cc:3037] T b4874706759a403992f9662c84a62b91 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.865032 25259 raft_consensus.cc:3037] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.866704 25123 raft_consensus.cc:2454] T b4874706759a403992f9662c84a62b91 P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 5.
I20240502 21:29:55.866828 25259 raft_consensus.cc:2396] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.867085 24789 leader_election.cc:302] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:55.867501 27483 raft_consensus.cc:2781] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:29:55.867697 27483 raft_consensus.cc:686] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [term 5 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:55.867744 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "acf184a70440492c80cfaea7d80323de"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:55.867954 24857 raft_consensus.cc:482] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.868059 24857 raft_consensus.cc:3037] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.868005 27483 consensus_queue.cc:237] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.871022 24857 raft_consensus.cc:504] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.871116 24706 catalog_manager.cc:5484] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 3 to 5, leader changed from 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 5 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:55.871781 24857 leader_election.cc:288] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 4 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:55.872133 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf184a70440492c80cfaea7d80323de" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.872323 25259 raft_consensus.cc:3037] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.872790 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c574664a57444c95b70d754cc46e313e"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:55.873005 24857 raft_consensus.cc:482] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.873131 24857 raft_consensus.cc:3037] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.873618 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf184a70440492c80cfaea7d80323de" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.874020 25123 raft_consensus.cc:3032] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:29:55.874137 25123 raft_consensus.cc:729] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:55.874456 25123 consensus_queue.cc:260] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.875025 25123 raft_consensus.cc:3037] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:29:55.875088 25259 raft_consensus.cc:2454] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 4.
I20240502 21:29:55.875635 24790 leader_election.cc:302] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:55.875656 24857 raft_consensus.cc:504] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.876055 27483 raft_consensus.cc:2781] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:29:55.876313 24857 leader_election.cc:288] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 5 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:55.876901 27483 raft_consensus.cc:686] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [term 4 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:55.877008 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c574664a57444c95b70d754cc46e313e" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.877221 24990 raft_consensus.cc:3032] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:29:55.877308 24990 raft_consensus.cc:729] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.877190 27483 consensus_queue.cc:237] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.877574 24990 consensus_queue.cc:260] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.877907 24990 raft_consensus.cc:3037] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.880270 24990 raft_consensus.cc:2396] T c574664a57444c95b70d754cc46e313e P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.881140 25124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c574664a57444c95b70d754cc46e313e" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.881341 25124 raft_consensus.cc:3037] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.881503 24706 catalog_manager.cc:5484] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 3 to 4, leader changed from 5bf90e514781418da921c046aa5d9ced (127.29.82.67) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 4 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } }
I20240502 21:29:55.882100 25123 raft_consensus.cc:2454] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 4.
I20240502 21:29:55.882088 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e145bbec053d4028ad7bd38d25dd4315"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:55.882277 24857 raft_consensus.cc:482] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.882391 24857 raft_consensus.cc:3037] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.884073 25124 raft_consensus.cc:2396] T c574664a57444c95b70d754cc46e313e P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.884480 24857 raft_consensus.cc:504] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.884557 24789 leader_election.cc:302] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a171ad73f86643afb1444248c0d0bc8b; no voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557
I20240502 21:29:55.885165 24857 leader_election.cc:288] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 5 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:55.885407 27483 raft_consensus.cc:2726] T c574664a57444c95b70d754cc46e313e P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240502 21:29:55.885927 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e145bbec053d4028ad7bd38d25dd4315" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.886101 25124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e145bbec053d4028ad7bd38d25dd4315" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.886174 25259 raft_consensus.cc:3037] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.886286 25124 raft_consensus.cc:3037] T e145bbec053d4028ad7bd38d25dd4315 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.886430 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fed2d260d00d470e95e9a62cc71ba94f"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:55.886617 24857 raft_consensus.cc:482] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.886744 24857 raft_consensus.cc:3037] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.888960 25124 raft_consensus.cc:2454] T e145bbec053d4028ad7bd38d25dd4315 P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 5.
I20240502 21:29:55.889027 25259 raft_consensus.cc:2396] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.889329 24789 leader_election.cc:302] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:55.889624 27483 raft_consensus.cc:2781] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:29:55.889811 27483 raft_consensus.cc:686] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [term 5 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:55.889912 24857 raft_consensus.cc:504] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.890184 27483 consensus_queue.cc:237] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.890916 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fed2d260d00d470e95e9a62cc71ba94f" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.891180 25259 raft_consensus.cc:3037] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.891259 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fed2d260d00d470e95e9a62cc71ba94f" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.891484 24990 raft_consensus.cc:3032] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:29:55.891590 24990 raft_consensus.cc:729] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.891862 24990 consensus_queue.cc:260] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.892172 24990 raft_consensus.cc:3037] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.895047 24990 raft_consensus.cc:2396] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:55.895211 25259 raft_consensus.cc:2454] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 6.
I20240502 21:29:55.895393 24857 leader_election.cc:288] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 6 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:55.895726 24790 leader_election.cc:302] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:55.896237 27306 raft_consensus.cc:2781] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:29:55.896425 27306 raft_consensus.cc:686] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [term 6 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:55.896780 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "89b06cd6b69d41d0b49ad40f605a3e16"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:55.896752 27306 consensus_queue.cc:237] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.896979 24857 raft_consensus.cc:482] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.897073 24857 raft_consensus.cc:3037] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.900247 24857 raft_consensus.cc:504] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.900915 24857 leader_election.cc:288] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 5 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:55.901389 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b06cd6b69d41d0b49ad40f605a3e16" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.901106 24706 catalog_manager.cc:5484] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 3 to 5, leader changed from 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 5 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:55.901693 25259 raft_consensus.cc:3037] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.902016 24855 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8423ff21d93741cdab3093ea454bb532"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:55.902180 24855 raft_consensus.cc:482] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.902272 24855 raft_consensus.cc:3037] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.903988 25259 raft_consensus.cc:2396] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.904655 25124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b06cd6b69d41d0b49ad40f605a3e16" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.904685 24855 raft_consensus.cc:504] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.904855 25124 raft_consensus.cc:3037] T 89b06cd6b69d41d0b49ad40f605a3e16 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.905242 24855 leader_election.cc:288] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 6 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:55.906220 24855 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "986c305824414299ac30f1e50f514c8a"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:55.906435 24855 raft_consensus.cc:482] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.906545 24855 raft_consensus.cc:3037] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.907338 25124 raft_consensus.cc:2454] T 89b06cd6b69d41d0b49ad40f605a3e16 P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 5.
I20240502 21:29:55.909225 24855 raft_consensus.cc:504] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.909619 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8423ff21d93741cdab3093ea454bb532" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.909875 24990 raft_consensus.cc:3032] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:29:55.909981 24990 raft_consensus.cc:729] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.910163 24855 leader_election.cc:288] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 6 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:55.910298 24990 consensus_queue.cc:260] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.910334 25258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c305824414299ac30f1e50f514c8a" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.910598 25258 raft_consensus.cc:3037] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.910612 24990 raft_consensus.cc:3037] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.913645 25258 raft_consensus.cc:2454] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 6.
I20240502 21:29:55.913755 24990 raft_consensus.cc:2396] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:55.914116 24790 leader_election.cc:302] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:55.914296 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "48e7c6534b544336ae216d3200f4f01f"
dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
from {username='slave'} at 127.0.0.1:49090
I20240502 21:29:55.914424 27483 raft_consensus.cc:2781] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:29:55.914520 24857 raft_consensus.cc:482] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.914597 27483 raft_consensus.cc:686] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 6 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:55.914681 24857 raft_consensus.cc:3037] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.914918 27483 consensus_queue.cc:237] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.915023 24706 catalog_manager.cc:5484] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 5 to 6, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 6 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:55.916160 24789 leader_election.cc:302] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b; no voters: 45b7d5b9696b4046b522c880c748fe6c
I20240502 21:29:55.916512 27306 raft_consensus.cc:2781] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:29:55.916682 27306 raft_consensus.cc:686] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [term 5 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:55.916994 27306 consensus_queue.cc:237] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.920672 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c305824414299ac30f1e50f514c8a" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.920930 24990 raft_consensus.cc:3032] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:29:55.921031 24990 raft_consensus.cc:729] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.922072 24857 raft_consensus.cc:504] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.922940 24990 consensus_queue.cc:260] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.923298 24990 raft_consensus.cc:3037] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.923343 25258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e7c6534b544336ae216d3200f4f01f" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.923679 25258 raft_consensus.cc:3037] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.927266 24990 raft_consensus.cc:2396] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:55.927326 25258 raft_consensus.cc:2454] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 6.
I20240502 21:29:55.927903 24790 leader_election.cc:302] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:55.927897 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8423ff21d93741cdab3093ea454bb532" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:55.928272 27306 raft_consensus.cc:2781] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:29:55.928651 27306 raft_consensus.cc:686] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [term 6 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:55.928107 25259 raft_consensus.cc:3037] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.929141 24857 leader_election.cc:288] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 6 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:55.929539 27306 consensus_queue.cc:237] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.931627 24706 catalog_manager.cc:5484] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 5 to 6, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 6 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:55.932170 24706 catalog_manager.cc:5484] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 3 to 5, leader changed from 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 5 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:55.933357 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e7c6534b544336ae216d3200f4f01f" candidate_uuid: "a171ad73f86643afb1444248c0d0bc8b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.933583 24990 raft_consensus.cc:3032] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:29:55.933674 24990 raft_consensus.cc:729] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:55.933979 24990 consensus_queue.cc:260] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.934332 24990 raft_consensus.cc:3037] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.937783 25259 raft_consensus.cc:2454] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a171ad73f86643afb1444248c0d0bc8b in term 6.
I20240502 21:29:55.938243 24790 leader_election.cc:302] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b; no voters: 64d97dd1674d43ccadd7b4a38eeff557
I20240502 21:29:55.938766 27483 raft_consensus.cc:2781] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:29:55.939201 27483 raft_consensus.cc:686] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 6 LEADER]: Becoming Leader. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:55.939606 27483 consensus_queue.cc:237] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.940735 24990 raft_consensus.cc:2396] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a171ad73f86643afb1444248c0d0bc8b for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:55.944301 24706 catalog_manager.cc:5484] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 5 to 6, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 6 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } }
I20240502 21:29:55.945240 25259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "87459ddf48034b28b7b3e0d0d7de91f1"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:55.945461 25259 raft_consensus.cc:482] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.945591 25259 raft_consensus.cc:3037] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.948101 25259 raft_consensus.cc:504] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.949025 25259 leader_election.cc:288] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 5 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:55.949813 25124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87459ddf48034b28b7b3e0d0d7de91f1" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.949808 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87459ddf48034b28b7b3e0d0d7de91f1" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.950007 25124 raft_consensus.cc:3037] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.950023 24990 raft_consensus.cc:3037] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.950331 24706 catalog_manager.cc:5484] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b reported cstate change: term changed from 5 to 6, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to a171ad73f86643afb1444248c0d0bc8b (127.29.82.65). New cstate: current_term: 6 leader_uuid: "a171ad73f86643afb1444248c0d0bc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:55.952718 24990 raft_consensus.cc:2454] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 5.
I20240502 21:29:55.953487 25259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4127d80443f1499ab9a30935bbd4ca8e"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:55.953684 25259 raft_consensus.cc:482] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.953821 25259 raft_consensus.cc:3037] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.953872 25191 leader_election.cc:302] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:55.954169 27358 raft_consensus.cc:2781] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:29:55.954403 25124 raft_consensus.cc:2396] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:29:55.956838 25259 raft_consensus.cc:504] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.957581 25259 leader_election.cc:288] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 6 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:55.958235 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4127d80443f1499ab9a30935bbd4ca8e" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.958106 25124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4127d80443f1499ab9a30935bbd4ca8e" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.958446 24990 raft_consensus.cc:3037] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.958488 25124 raft_consensus.cc:3037] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.958724 25259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "22e48679f46b4b72aa0f0736c76be6f1"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:55.959023 25259 raft_consensus.cc:482] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.959208 25259 raft_consensus.cc:3037] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.961232 24990 raft_consensus.cc:2454] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 6.
I20240502 21:29:55.961319 25124 raft_consensus.cc:2396] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240502 21:29:55.962149 25191 leader_election.cc:302] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:55.962779 27504 raft_consensus.cc:2781] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:29:55.962924 27504 raft_consensus.cc:686] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [term 6 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:55.963234 27504 consensus_queue.cc:237] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.963188 25259 raft_consensus.cc:504] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.964816 25259 leader_election.cc:288] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 5 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:55.965750 25259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d643c15afbec4933a0d0a145f17f703c"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:55.966054 25259 raft_consensus.cc:482] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.966298 25259 raft_consensus.cc:3037] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.967770 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22e48679f46b4b72aa0f0736c76be6f1" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.968008 24857 raft_consensus.cc:3032] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:29:55.968103 24857 raft_consensus.cc:729] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:55.968077 25124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22e48679f46b4b72aa0f0736c76be6f1" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.968314 25124 raft_consensus.cc:3037] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.968410 24857 consensus_queue.cc:260] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.968729 24857 raft_consensus.cc:3037] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:55.970309 25259 raft_consensus.cc:504] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.970943 25259 leader_election.cc:288] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 6 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:55.971930 25259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "083a9351f8ee4a44a86cb2d42f1ec558"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:55.972142 24857 raft_consensus.cc:2396] T 22e48679f46b4b72aa0f0736c76be6f1 P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240502 21:29:55.972180 25259 raft_consensus.cc:482] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.972393 25259 raft_consensus.cc:3037] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.972949 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d643c15afbec4933a0d0a145f17f703c" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.973181 25123 raft_consensus.cc:3037] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.973183 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d643c15afbec4933a0d0a145f17f703c" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:55.973400 24990 raft_consensus.cc:3037] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.975152 24990 raft_consensus.cc:2454] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 6.
I20240502 21:29:55.975550 25191 leader_election.cc:302] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:55.975847 25124 raft_consensus.cc:2454] T 22e48679f46b4b72aa0f0736c76be6f1 P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 5.
I20240502 21:29:55.975873 27504 raft_consensus.cc:2781] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:29:55.976016 27504 raft_consensus.cc:686] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [term 6 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:55.976269 25191 leader_election.cc:302] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced; no voters: a171ad73f86643afb1444248c0d0bc8b
I20240502 21:29:55.976349 25123 raft_consensus.cc:2396] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240502 21:29:55.977036 27504 consensus_queue.cc:237] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.980058 25259 raft_consensus.cc:504] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.980967 25259 leader_election.cc:288] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 6 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:55.982101 25259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bdecfa03390e405ca4925f8d1a40e0ca"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:55.982398 25259 raft_consensus.cc:482] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.982628 25259 raft_consensus.cc:3037] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:29:55.984961 27504 raft_consensus.cc:2781] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:29:55.985101 27504 raft_consensus.cc:686] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [term 5 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:55.985410 27504 consensus_queue.cc:237] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.986510 25259 raft_consensus.cc:504] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.987273 25259 leader_election.cc:288] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 7 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:55.988389 25259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5360f56e93ef486b94bebcddf2edfaaa"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:55.988647 25259 raft_consensus.cc:482] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.988777 25259 raft_consensus.cc:3037] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.992141 27358 raft_consensus.cc:686] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [term 5 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:55.992165 25259 raft_consensus.cc:504] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:55.992466 27358 consensus_queue.cc:237] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:55.992561 25124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "083a9351f8ee4a44a86cb2d42f1ec558" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:55.992785 25259 leader_election.cc:288] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 6 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:55.992864 25124 raft_consensus.cc:3037] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.993939 25259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57b40e72e8324b4ba6febf5eda24d333"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:55.994138 25259 raft_consensus.cc:482] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.994275 25259 raft_consensus.cc:3037] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.995797 25124 raft_consensus.cc:2454] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 6.
I20240502 21:29:55.996430 24706 catalog_manager.cc:5484] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 3 to 5, leader changed from 5bf90e514781418da921c046aa5d9ced (127.29.82.67) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 5 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:55.996738 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "083a9351f8ee4a44a86cb2d42f1ec558" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:55.996963 24857 raft_consensus.cc:3032] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:29:55.997045 24857 raft_consensus.cc:729] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:55.997279 25259 raft_consensus.cc:504] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:55.997395 24857 consensus_queue.cc:260] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:55.997727 24857 raft_consensus.cc:3037] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:55.997860 25259 leader_election.cc:288] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 6 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:55.998826 25259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "69afb121cb564de3a9bec5e3f1cc0492"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:55.999089 25259 raft_consensus.cc:482] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:55.999307 25259 raft_consensus.cc:3037] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:56.001096 24857 raft_consensus.cc:2396] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:56.002177 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdecfa03390e405ca4925f8d1a40e0ca" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:56.002244 25259 raft_consensus.cc:504] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:56.002444 24990 raft_consensus.cc:3037] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:29:56.002482 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdecfa03390e405ca4925f8d1a40e0ca" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:56.002763 24857 raft_consensus.cc:3032] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:29:56.002862 24857 raft_consensus.cc:729] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:56.003063 25259 leader_election.cc:288] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 6 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:56.003152 24857 consensus_queue.cc:260] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:56.003479 24857 raft_consensus.cc:3037] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:29:56.003074 24706 catalog_manager.cc:5484] T 22e48679f46b4b72aa0f0736c76be6f1 P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 4 to 5, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 5 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:56.004163 25259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "89c4eee02b444dc188172fd8c7b411bb"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:56.004377 25259 raft_consensus.cc:482] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:56.004309 24706 catalog_manager.cc:5484] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 4 to 6, leader changed from 5bf90e514781418da921c046aa5d9ced (127.29.82.67) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 6 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:56.004539 25259 raft_consensus.cc:3037] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:56.004765 24706 catalog_manager.cc:5484] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 4 to 6, leader changed from 5bf90e514781418da921c046aa5d9ced (127.29.82.67) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 6 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:56.005302 24990 raft_consensus.cc:2396] T bdecfa03390e405ca4925f8d1a40e0ca P 64d97dd1674d43ccadd7b4a38eeff557 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:56.006286 24857 raft_consensus.cc:2396] T bdecfa03390e405ca4925f8d1a40e0ca P a171ad73f86643afb1444248c0d0bc8b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:56.006791 25191 leader_election.cc:302] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced; no voters:
I20240502 21:29:56.007092 27358 raft_consensus.cc:2781] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:29:56.007259 27358 raft_consensus.cc:686] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 6 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:56.007457 25259 raft_consensus.cc:504] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:56.007565 27358 consensus_queue.cc:237] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:56.008186 25259 leader_election.cc:288] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 6 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:56.009650 25259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b4874706759a403992f9662c84a62b91"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:56.009948 25259 raft_consensus.cc:482] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:56.010069 25259 raft_consensus.cc:3037] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:56.010360 24855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5360f56e93ef486b94bebcddf2edfaaa" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:56.010566 24855 raft_consensus.cc:3032] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:29:56.010691 24855 raft_consensus.cc:729] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:56.010957 24855 consensus_queue.cc:260] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:56.011200 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b40e72e8324b4ba6febf5eda24d333" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:56.011353 24855 raft_consensus.cc:3037] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:56.011415 24856 raft_consensus.cc:3032] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:29:56.011523 24856 raft_consensus.cc:729] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:56.011898 25191 leader_election.cc:302] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c; no voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b
I20240502 21:29:56.011828 24856 consensus_queue.cc:260] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:56.012136 24856 raft_consensus.cc:3037] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:56.012779 25259 raft_consensus.cc:504] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:56.013353 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69afb121cb564de3a9bec5e3f1cc0492" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:56.013456 25259 leader_election.cc:288] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 6 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:56.013561 24991 raft_consensus.cc:3037] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:56.015822 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c4eee02b444dc188172fd8c7b411bb" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:56.016039 24992 raft_consensus.cc:3037] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:56.016527 24991 raft_consensus.cc:2454] T 69afb121cb564de3a9bec5e3f1cc0492 P 64d97dd1674d43ccadd7b4a38eeff557 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 6.
I20240502 21:29:56.017740 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69afb121cb564de3a9bec5e3f1cc0492" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:56.018026 25123 raft_consensus.cc:3037] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:56.018899 24992 raft_consensus.cc:2454] T 89c4eee02b444dc188172fd8c7b411bb P 64d97dd1674d43ccadd7b4a38eeff557 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 6.
I20240502 21:29:56.019275 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b40e72e8324b4ba6febf5eda24d333" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:56.019465 24990 raft_consensus.cc:3037] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:56.020927 25123 raft_consensus.cc:2396] T 69afb121cb564de3a9bec5e3f1cc0492 P 5bf90e514781418da921c046aa5d9ced [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240502 21:29:56.021276 25121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c4eee02b444dc188172fd8c7b411bb" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:56.021427 25121 raft_consensus.cc:3037] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:56.022341 24990 raft_consensus.cc:2396] T 57b40e72e8324b4ba6febf5eda24d333 P 64d97dd1674d43ccadd7b4a38eeff557 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:56.024363 27358 raft_consensus.cc:2726] T bdecfa03390e405ca4925f8d1a40e0ca P 45b7d5b9696b4046b522c880c748fe6c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240502 21:29:56.024571 24855 raft_consensus.cc:2396] T 5360f56e93ef486b94bebcddf2edfaaa P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:56.025048 24856 raft_consensus.cc:2396] T 57b40e72e8324b4ba6febf5eda24d333 P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:56.025013 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4874706759a403992f9662c84a62b91" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:56.025233 24857 raft_consensus.cc:3032] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:29:56.025315 24857 raft_consensus.cc:729] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:56.025688 25121 raft_consensus.cc:2396] T 89c4eee02b444dc188172fd8c7b411bb P 5bf90e514781418da921c046aa5d9ced [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240502 21:29:56.025642 24857 consensus_queue.cc:260] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:56.025944 24857 raft_consensus.cc:3037] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:56.027197 24706 catalog_manager.cc:5484] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 5 to 6, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 6 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } }
I20240502 21:29:56.028556 25124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5360f56e93ef486b94bebcddf2edfaaa" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:56.028792 25124 raft_consensus.cc:3037] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:56.031764 25124 raft_consensus.cc:2454] T 5360f56e93ef486b94bebcddf2edfaaa P 5bf90e514781418da921c046aa5d9ced [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 6.
I20240502 21:29:56.032066 25259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "acf184a70440492c80cfaea7d80323de"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:56.032327 25191 leader_election.cc:302] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters: 5bf90e514781418da921c046aa5d9ced
I20240502 21:29:56.032387 25259 raft_consensus.cc:482] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:56.032640 27358 raft_consensus.cc:2781] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:29:56.032830 25191 leader_election.cc:302] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters: 5bf90e514781418da921c046aa5d9ced
I20240502 21:29:56.033018 27358 raft_consensus.cc:686] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [term 6 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:56.033320 27358 consensus_queue.cc:237] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:56.033510 25191 leader_election.cc:302] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c; no voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b
I20240502 21:29:56.033788 27507 raft_consensus.cc:2781] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:29:56.033946 27507 raft_consensus.cc:686] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [term 6 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:56.034265 27507 consensus_queue.cc:237] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:56.034610 27502 raft_consensus.cc:2726] T 57b40e72e8324b4ba6febf5eda24d333 P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240502 21:29:56.037945 25259 raft_consensus.cc:3037] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:56.031873 25121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4874706759a403992f9662c84a62b91" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:56.039124 25121 raft_consensus.cc:3037] T b4874706759a403992f9662c84a62b91 P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:56.041759 25121 raft_consensus.cc:2454] T b4874706759a403992f9662c84a62b91 P 5bf90e514781418da921c046aa5d9ced [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 6.
I20240502 21:29:56.042348 25191 leader_election.cc:302] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced; no voters:
I20240502 21:29:56.042662 27507 raft_consensus.cc:2781] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:29:56.043012 27507 raft_consensus.cc:686] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [term 6 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:56.043316 27507 consensus_queue.cc:237] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:56.044600 25259 raft_consensus.cc:504] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:56.045236 25259 leader_election.cc:288] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 5 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:56.047134 25121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf184a70440492c80cfaea7d80323de" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:56.047399 25121 raft_consensus.cc:3037] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:56.047209 24706 catalog_manager.cc:5484] T 69afb121cb564de3a9bec5e3f1cc0492 P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 4 to 6, leader changed from 5bf90e514781418da921c046aa5d9ced (127.29.82.67) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 6 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } }
I20240502 21:29:56.050241 25121 raft_consensus.cc:2454] T acf184a70440492c80cfaea7d80323de P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 5.
I20240502 21:29:56.050956 25191 leader_election.cc:302] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced; no voters:
I20240502 21:29:56.051247 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf184a70440492c80cfaea7d80323de" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:56.051484 24856 raft_consensus.cc:3032] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:29:56.051611 24856 raft_consensus.cc:729] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:56.051896 24856 consensus_queue.cc:260] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:56.052196 24856 raft_consensus.cc:3037] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:56.052747 27358 raft_consensus.cc:2781] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:29:56.053050 27358 raft_consensus.cc:686] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [term 5 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:56.053417 27358 consensus_queue.cc:237] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:56.054836 25191 leader_election.cc:302] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced; no voters: a171ad73f86643afb1444248c0d0bc8b
I20240502 21:29:56.055955 27358 raft_consensus.cc:2781] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:29:56.056380 27358 raft_consensus.cc:686] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [term 6 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:56.056766 27358 consensus_queue.cc:237] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:56.059486 24857 raft_consensus.cc:2396] T b4874706759a403992f9662c84a62b91 P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:56.060338 25259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a851d0bede1444c8a96ea45ea948bd59"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:56.061013 25259 raft_consensus.cc:482] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:56.061432 25259 raft_consensus.cc:3037] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:29:56.061998 24856 raft_consensus.cc:2396] T acf184a70440492c80cfaea7d80323de P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:56.065439 24706 catalog_manager.cc:5484] T 89c4eee02b444dc188172fd8c7b411bb P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 4 to 6, leader changed from 5bf90e514781418da921c046aa5d9ced (127.29.82.67) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 6 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:56.065805 25259 raft_consensus.cc:504] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:56.066023 24706 catalog_manager.cc:5484] T b4874706759a403992f9662c84a62b91 P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 5 to 6, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 6 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:56.066525 25259 leader_election.cc:288] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 7 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:56.066927 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851d0bede1444c8a96ea45ea948bd59" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:56.067121 24990 raft_consensus.cc:3037] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:29:56.069991 24990 raft_consensus.cc:2396] T a851d0bede1444c8a96ea45ea948bd59 P 64d97dd1674d43ccadd7b4a38eeff557 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:56.070915 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851d0bede1444c8a96ea45ea948bd59" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:56.071102 24856 raft_consensus.cc:3032] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:29:56.071185 24856 raft_consensus.cc:729] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:56.072414 25259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e145bbec053d4028ad7bd38d25dd4315"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:56.072616 25259 raft_consensus.cc:482] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:56.071427 24856 consensus_queue.cc:260] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:56.072944 24856 raft_consensus.cc:3037] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:29:56.073169 25259 raft_consensus.cc:3037] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:56.074088 24707 catalog_manager.cc:5484] T acf184a70440492c80cfaea7d80323de P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 4 to 5, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 5 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } }
I20240502 21:29:56.074611 24707 catalog_manager.cc:5484] T 5360f56e93ef486b94bebcddf2edfaaa P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 5 to 6, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 6 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:56.075901 24856 raft_consensus.cc:2396] T a851d0bede1444c8a96ea45ea948bd59 P a171ad73f86643afb1444248c0d0bc8b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:56.076550 25191 leader_election.cc:302] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c; no voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b
I20240502 21:29:56.076897 27507 raft_consensus.cc:2726] T a851d0bede1444c8a96ea45ea948bd59 P 45b7d5b9696b4046b522c880c748fe6c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240502 21:29:56.080050 25259 raft_consensus.cc:504] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:56.080744 25259 leader_election.cc:288] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 6 election: Requested vote from peers 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:56.081349 25124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e145bbec053d4028ad7bd38d25dd4315" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:56.081607 25124 raft_consensus.cc:3037] T e145bbec053d4028ad7bd38d25dd4315 P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:56.081840 25259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fed2d260d00d470e95e9a62cc71ba94f"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:56.082043 25259 raft_consensus.cc:482] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:56.082160 25259 raft_consensus.cc:3037] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:29:56.084546 25124 raft_consensus.cc:2454] T e145bbec053d4028ad7bd38d25dd4315 P 5bf90e514781418da921c046aa5d9ced [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 6.
I20240502 21:29:56.084954 25259 raft_consensus.cc:504] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:56.085649 25259 leader_election.cc:288] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 7 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:56.085700 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e145bbec053d4028ad7bd38d25dd4315" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:56.085927 24856 raft_consensus.cc:3032] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:29:56.086035 24856 raft_consensus.cc:729] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:56.086336 24856 consensus_queue.cc:260] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:56.086683 24856 raft_consensus.cc:3037] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:56.086712 25258 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f25dfc6b0c04d878cd3db58d37cddec"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:56.086881 25258 raft_consensus.cc:482] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:56.087000 25258 raft_consensus.cc:3037] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:56.089413 24856 raft_consensus.cc:2396] T e145bbec053d4028ad7bd38d25dd4315 P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:56.089514 25258 raft_consensus.cc:504] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:56.091142 25258 leader_election.cc:288] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 6 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:56.090683 25261 raft_consensus.cc:1261] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Refusing update from remote peer 64d97dd1674d43ccadd7b4a38eeff557: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240502 21:29:56.091871 27446 consensus_queue.cc:1035] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:56.092451 25261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "69df123d92d74693b4654c6a809bbe89"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:56.092656 25261 raft_consensus.cc:482] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:56.092765 25261 raft_consensus.cc:3037] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:56.092828 25191 leader_election.cc:302] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced; no voters:
I20240502 21:29:56.093166 27507 raft_consensus.cc:2781] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:29:56.093392 27507 raft_consensus.cc:686] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [term 6 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:56.093780 27507 consensus_queue.cc:237] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:56.096148 25261 raft_consensus.cc:504] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:56.096743 25261 leader_election.cc:288] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 5 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:56.097631 25258 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "89b06cd6b69d41d0b49ad40f605a3e16"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:56.097874 25258 raft_consensus.cc:482] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:56.098033 25258 raft_consensus.cc:3037] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:56.097916 24711 catalog_manager.cc:5484] T e145bbec053d4028ad7bd38d25dd4315 P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 5 to 6, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 6 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:56.096107 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fed2d260d00d470e95e9a62cc71ba94f" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:56.099761 24856 raft_consensus.cc:3032] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:29:56.099910 24856 raft_consensus.cc:729] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:56.100281 24856 consensus_queue.cc:260] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:56.100725 24856 raft_consensus.cc:3037] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:29:56.104655 25258 raft_consensus.cc:504] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:56.105352 24856 raft_consensus.cc:2396] T fed2d260d00d470e95e9a62cc71ba94f P a171ad73f86643afb1444248c0d0bc8b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:56.105928 25258 leader_election.cc:288] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 6 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:56.106185 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fed2d260d00d470e95e9a62cc71ba94f" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:56.106495 24990 raft_consensus.cc:3037] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:29:56.106948 25261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "52153a02d80043f0a1840b0e5aadd9b3"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:56.107159 25261 raft_consensus.cc:482] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:56.107318 25261 raft_consensus.cc:3037] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:56.109660 24990 raft_consensus.cc:2396] T fed2d260d00d470e95e9a62cc71ba94f P 64d97dd1674d43ccadd7b4a38eeff557 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:56.109699 25124 raft_consensus.cc:1261] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Refusing update from remote peer 64d97dd1674d43ccadd7b4a38eeff557: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240502 21:29:56.110433 26535 consensus_queue.cc:1035] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:56.111111 25261 raft_consensus.cc:504] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:56.111613 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f25dfc6b0c04d878cd3db58d37cddec" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:56.111810 24990 raft_consensus.cc:3037] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:56.111806 25261 leader_election.cc:288] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 5 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 5bf90e514781418da921c046aa5d9ced (127.29.82.67:37875)
I20240502 21:29:56.112780 25260 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "986c305824414299ac30f1e50f514c8a"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:56.112941 25260 raft_consensus.cc:482] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:56.113041 25260 raft_consensus.cc:3037] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:29:56.114171 25259 raft_consensus.cc:1226] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Rejecting Update request from peer 64d97dd1674d43ccadd7b4a38eeff557 for earlier term 4. Current term is 5. Ops: []
I20240502 21:29:56.114753 24990 raft_consensus.cc:2454] T 6f25dfc6b0c04d878cd3db58d37cddec P 64d97dd1674d43ccadd7b4a38eeff557 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 6.
I20240502 21:29:56.114722 26535 consensus_queue.cc:1046] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:56.115025 26535 raft_consensus.cc:3032] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:29:56.115101 26535 raft_consensus.cc:729] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:56.115367 26535 consensus_queue.cc:260] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:56.115561 25121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f25dfc6b0c04d878cd3db58d37cddec" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:56.115808 26535 raft_consensus.cc:3037] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:56.115816 25121 raft_consensus.cc:3037] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:56.116077 25260 raft_consensus.cc:504] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:56.116657 25260 leader_election.cc:288] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 7 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:56.117611 25260 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8423ff21d93741cdab3093ea454bb532"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:56.117844 25260 raft_consensus.cc:482] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:56.118009 25260 raft_consensus.cc:3037] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:29:56.119594 25121 raft_consensus.cc:2396] T 6f25dfc6b0c04d878cd3db58d37cddec P 5bf90e514781418da921c046aa5d9ced [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240502 21:29:56.120738 25260 raft_consensus.cc:504] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:56.121066 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69df123d92d74693b4654c6a809bbe89" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:56.121361 24990 raft_consensus.cc:3032] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:29:56.121431 25260 leader_election.cc:288] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 7 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:56.121516 24990 raft_consensus.cc:729] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 64d97dd1674d43ccadd7b4a38eeff557, State: Running, Role: LEADER
I20240502 21:29:56.121917 24990 consensus_queue.cc:260] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:56.122304 24990 raft_consensus.cc:3037] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:56.122623 25259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "48e7c6534b544336ae216d3200f4f01f"
dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
from {username='slave'} at 127.0.0.1:58114
I20240502 21:29:56.122792 25259 raft_consensus.cc:482] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:56.122891 25259 raft_consensus.cc:3037] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:29:56.124308 25121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69df123d92d74693b4654c6a809bbe89" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:56.124588 25121 raft_consensus.cc:3037] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:56.125288 25259 raft_consensus.cc:504] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:56.125838 25259 leader_election.cc:288] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 7 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:56.128242 25121 raft_consensus.cc:2454] T 69df123d92d74693b4654c6a809bbe89 P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 5.
I20240502 21:29:56.129894 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b06cd6b69d41d0b49ad40f605a3e16" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:56.130211 24856 raft_consensus.cc:3032] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:29:56.130347 24856 raft_consensus.cc:729] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:56.132180 25121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b06cd6b69d41d0b49ad40f605a3e16" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:56.132454 25121 raft_consensus.cc:3037] T 89b06cd6b69d41d0b49ad40f605a3e16 P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:56.133975 25124 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4127d80443f1499ab9a30935bbd4ca8e"
dest_uuid: "5bf90e514781418da921c046aa5d9ced"
from {username='slave'} at 127.0.0.1:52128
I20240502 21:29:56.134213 25124 raft_consensus.cc:482] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:56.134327 25124 raft_consensus.cc:3037] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:29:56.135303 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52153a02d80043f0a1840b0e5aadd9b3" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:56.136869 25124 raft_consensus.cc:504] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:56.138263 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4127d80443f1499ab9a30935bbd4ca8e" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:56.138461 25259 raft_consensus.cc:3032] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:29:56.138561 25259 raft_consensus.cc:729] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:56.138746 25124 leader_election.cc:288] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 7 election: Requested vote from peers 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721), 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099)
I20240502 21:29:56.139133 24992 raft_consensus.cc:2454] T 52153a02d80043f0a1840b0e5aadd9b3 P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 5.
I20240502 21:29:56.139590 25123 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f801dcce3bfc42bf89a41156826a9203"
dest_uuid: "5bf90e514781418da921c046aa5d9ced"
from {username='slave'} at 127.0.0.1:52128
I20240502 21:29:56.139768 25123 raft_consensus.cc:482] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:56.139786 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4127d80443f1499ab9a30935bbd4ca8e" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:56.139886 25123 raft_consensus.cc:3037] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:56.140074 24991 raft_consensus.cc:3037] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:29:56.142375 25123 raft_consensus.cc:504] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:56.143343 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f801dcce3bfc42bf89a41156826a9203" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:56.143620 24857 raft_consensus.cc:3037] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:56.143729 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f801dcce3bfc42bf89a41156826a9203" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:56.143888 24989 raft_consensus.cc:3037] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:56.143539 24991 raft_consensus.cc:2454] T 4127d80443f1499ab9a30935bbd4ca8e P 64d97dd1674d43ccadd7b4a38eeff557 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 7.
I20240502 21:29:56.144299 24856 consensus_queue.cc:260] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:56.144624 24856 raft_consensus.cc:3037] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:56.146531 24857 raft_consensus.cc:2454] T f801dcce3bfc42bf89a41156826a9203 P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 6.
I20240502 21:29:56.146670 24989 raft_consensus.cc:2396] T f801dcce3bfc42bf89a41156826a9203 P 64d97dd1674d43ccadd7b4a38eeff557 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5bf90e514781418da921c046aa5d9ced for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240502 21:29:56.147322 25058 leader_election.cc:302] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:56.147632 27414 raft_consensus.cc:2781] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:29:56.147753 27414 raft_consensus.cc:686] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [term 6 LEADER]: Becoming Leader. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:56.147922 25120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52153a02d80043f0a1840b0e5aadd9b3" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "5bf90e514781418da921c046aa5d9ced"
I20240502 21:29:56.148008 27414 consensus_queue.cc:237] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:56.148169 25120 raft_consensus.cc:3037] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:29:56.149704 25058 leader_election.cc:302] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557; no voters:
I20240502 21:29:56.150100 27408 raft_consensus.cc:2781] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 7 FOLLOWER]: Leader election won for term 7
I20240502 21:29:56.150228 27408 raft_consensus.cc:686] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [term 7 LEADER]: Becoming Leader. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:56.150472 27408 consensus_queue.cc:237] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:56.151540 25120 raft_consensus.cc:2454] T 52153a02d80043f0a1840b0e5aadd9b3 P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 5.
I20240502 21:29:56.152567 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c305824414299ac30f1e50f514c8a" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:56.152772 24857 raft_consensus.cc:3032] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:29:56.152877 24857 raft_consensus.cc:729] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:56.153007 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c305824414299ac30f1e50f514c8a" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:56.153226 24989 raft_consensus.cc:3037] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:29:56.153129 24857 consensus_queue.cc:260] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:56.153436 24857 raft_consensus.cc:3037] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:29:56.153672 24711 catalog_manager.cc:5484] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced reported cstate change: term changed from 4 to 6, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to 5bf90e514781418da921c046aa5d9ced (127.29.82.67). New cstate: current_term: 6 leader_uuid: "5bf90e514781418da921c046aa5d9ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:56.156206 25123 leader_election.cc:288] T f801dcce3bfc42bf89a41156826a9203 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 6 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:56.156450 24989 raft_consensus.cc:2396] T 986c305824414299ac30f1e50f514c8a P 64d97dd1674d43ccadd7b4a38eeff557 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:56.157138 25191 leader_election.cc:302] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced; no voters:
I20240502 21:29:56.157173 25120 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "87459ddf48034b28b7b3e0d0d7de91f1"
dest_uuid: "5bf90e514781418da921c046aa5d9ced"
from {username='slave'} at 127.0.0.1:52128
I20240502 21:29:56.157338 25120 raft_consensus.cc:482] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:56.157907 25120 raft_consensus.cc:3037] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:56.158603 24857 raft_consensus.cc:2396] T 986c305824414299ac30f1e50f514c8a P a171ad73f86643afb1444248c0d0bc8b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:56.158627 24990 raft_consensus.cc:2396] T 69df123d92d74693b4654c6a809bbe89 P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240502 21:29:56.157676 25259 consensus_queue.cc:260] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:56.159129 25259 raft_consensus.cc:3037] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:29:56.160876 24711 catalog_manager.cc:5484] T 4127d80443f1499ab9a30935bbd4ca8e P 5bf90e514781418da921c046aa5d9ced reported cstate change: term changed from 6 to 7, leader changed from 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68) to 5bf90e514781418da921c046aa5d9ced (127.29.82.67). New cstate: current_term: 7 leader_uuid: "5bf90e514781418da921c046aa5d9ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:56.163128 25259 raft_consensus.cc:2396] T 4127d80443f1499ab9a30935bbd4ca8e P 45b7d5b9696b4046b522c880c748fe6c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5bf90e514781418da921c046aa5d9ced for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240502 21:29:56.163718 27507 raft_consensus.cc:2781] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:29:56.163868 27507 raft_consensus.cc:686] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [term 5 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:56.164191 27507 consensus_queue.cc:237] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:56.165800 25191 leader_election.cc:302] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced; no voters:
I20240502 21:29:56.167016 27358 raft_consensus.cc:2781] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:29:56.167147 27358 raft_consensus.cc:686] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [term 5 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:56.166908 24711 catalog_manager.cc:5484] T 69df123d92d74693b4654c6a809bbe89 P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 4 to 5, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 5 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } }
I20240502 21:29:56.167371 27358 consensus_queue.cc:237] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:56.169224 25191 leader_election.cc:302] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c; no voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b
I20240502 21:29:56.170804 27507 raft_consensus.cc:2726] T fed2d260d00d470e95e9a62cc71ba94f P 45b7d5b9696b4046b522c880c748fe6c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240502 21:29:56.171293 25191 leader_election.cc:302] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 64d97dd1674d43ccadd7b4a38eeff557; no voters: 5bf90e514781418da921c046aa5d9ced
I20240502 21:29:56.172237 24707 catalog_manager.cc:5484] T 52153a02d80043f0a1840b0e5aadd9b3 P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 4 to 5, leader changed from 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 5 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:56.173163 25121 raft_consensus.cc:2454] T 89b06cd6b69d41d0b49ad40f605a3e16 P 5bf90e514781418da921c046aa5d9ced [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b7d5b9696b4046b522c880c748fe6c in term 6.
I20240502 21:29:56.173667 27507 raft_consensus.cc:2781] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:29:56.173830 27507 raft_consensus.cc:686] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [term 6 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:56.174105 27507 consensus_queue.cc:237] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:56.177534 25120 raft_consensus.cc:504] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:56.179378 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87459ddf48034b28b7b3e0d0d7de91f1" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:56.179600 25259 raft_consensus.cc:3032] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:29:56.179710 25259 raft_consensus.cc:729] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:56.179924 25120 leader_election.cc:288] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 6 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:56.179956 25259 consensus_queue.cc:260] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:56.180274 25259 raft_consensus.cc:3037] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:56.181061 25123 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3c1ce7cf99164a9a848cbb22ad2052ea"
dest_uuid: "5bf90e514781418da921c046aa5d9ced"
from {username='slave'} at 127.0.0.1:52128
I20240502 21:29:56.181157 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8423ff21d93741cdab3093ea454bb532" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:56.181305 25123 raft_consensus.cc:482] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:56.181468 24857 raft_consensus.cc:3032] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:29:56.181531 25123 raft_consensus.cc:3037] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:29:56.181602 24857 raft_consensus.cc:729] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:56.181943 24857 consensus_queue.cc:260] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } }
I20240502 21:29:56.182353 24857 raft_consensus.cc:3037] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:29:56.183408 25259 raft_consensus.cc:2396] T 87459ddf48034b28b7b3e0d0d7de91f1 P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5bf90e514781418da921c046aa5d9ced for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:56.184633 25123 raft_consensus.cc:504] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:56.185385 24857 raft_consensus.cc:2396] T 8423ff21d93741cdab3093ea454bb532 P a171ad73f86643afb1444248c0d0bc8b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:56.185532 24855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1ce7cf99164a9a848cbb22ad2052ea" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:56.185726 24855 raft_consensus.cc:3037] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:29:56.185936 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87459ddf48034b28b7b3e0d0d7de91f1" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:56.186180 24990 raft_consensus.cc:3037] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:29:56.186641 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1ce7cf99164a9a848cbb22ad2052ea" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:56.186836 24989 raft_consensus.cc:3037] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:29:56.188063 25191 leader_election.cc:302] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c; no voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b
I20240502 21:29:56.188406 24855 raft_consensus.cc:2454] T 3c1ce7cf99164a9a848cbb22ad2052ea P a171ad73f86643afb1444248c0d0bc8b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 7.
I20240502 21:29:56.188593 25191 leader_election.cc:302] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c, 5bf90e514781418da921c046aa5d9ced; no voters:
I20240502 21:29:56.188634 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8423ff21d93741cdab3093ea454bb532" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:56.188791 25058 leader_election.cc:302] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, a171ad73f86643afb1444248c0d0bc8b; no voters:
I20240502 21:29:56.188812 24991 raft_consensus.cc:3037] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:29:56.189074 27408 raft_consensus.cc:2781] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 7 FOLLOWER]: Leader election won for term 7
I20240502 21:29:56.189249 27408 raft_consensus.cc:686] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [term 7 LEADER]: Becoming Leader. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:56.189530 27408 consensus_queue.cc:237] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:56.190881 25123 leader_election.cc:288] T 3c1ce7cf99164a9a848cbb22ad2052ea P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 7 election: Requested vote from peers a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021), 64d97dd1674d43ccadd7b4a38eeff557 (127.29.82.66:39721)
I20240502 21:29:56.192085 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e7c6534b544336ae216d3200f4f01f" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:56.192214 25124 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "083a9351f8ee4a44a86cb2d42f1ec558"
dest_uuid: "5bf90e514781418da921c046aa5d9ced"
from {username='slave'} at 127.0.0.1:52128
I20240502 21:29:56.192304 24992 raft_consensus.cc:3037] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:29:56.192451 25124 raft_consensus.cc:482] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:56.192576 27358 raft_consensus.cc:2726] T 986c305824414299ac30f1e50f514c8a P 45b7d5b9696b4046b522c880c748fe6c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240502 21:29:56.192602 25124 raft_consensus.cc:3037] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:29:56.192759 27358 raft_consensus.cc:2781] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:29:56.192885 27358 raft_consensus.cc:686] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [term 6 LEADER]: Becoming Leader. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:56.193209 27358 consensus_queue.cc:237] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:56.195161 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e7c6534b544336ae216d3200f4f01f" candidate_uuid: "45b7d5b9696b4046b522c880c748fe6c" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:56.195381 24857 raft_consensus.cc:3032] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:29:56.195480 24857 raft_consensus.cc:729] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a171ad73f86643afb1444248c0d0bc8b, State: Running, Role: LEADER
I20240502 21:29:56.195847 24857 consensus_queue.cc:260] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:56.196213 24857 raft_consensus.cc:3037] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:29:56.196557 25124 raft_consensus.cc:504] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:56.199318 24991 raft_consensus.cc:2396] T 8423ff21d93741cdab3093ea454bb532 P 64d97dd1674d43ccadd7b4a38eeff557 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:56.202298 24992 raft_consensus.cc:2396] T 48e7c6534b544336ae216d3200f4f01f P 64d97dd1674d43ccadd7b4a38eeff557 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:56.204974 24857 raft_consensus.cc:2396] T 48e7c6534b544336ae216d3200f4f01f P a171ad73f86643afb1444248c0d0bc8b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:56.205262 24989 raft_consensus.cc:2396] T 3c1ce7cf99164a9a848cbb22ad2052ea P 64d97dd1674d43ccadd7b4a38eeff557 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5bf90e514781418da921c046aa5d9ced for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240502 21:29:56.206943 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "083a9351f8ee4a44a86cb2d42f1ec558" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:56.207180 25259 raft_consensus.cc:3032] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:29:56.207307 25259 raft_consensus.cc:729] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:56.207192 24855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "083a9351f8ee4a44a86cb2d42f1ec558" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a171ad73f86643afb1444248c0d0bc8b"
I20240502 21:29:56.207504 24855 raft_consensus.cc:3037] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:29:56.207556 25259 consensus_queue.cc:260] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } }
I20240502 21:29:56.207916 25259 raft_consensus.cc:3037] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:29:56.208909 25191 leader_election.cc:302] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c; no voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b
I20240502 21:29:56.209333 24856 raft_consensus.cc:2396] T 89b06cd6b69d41d0b49ad40f605a3e16 P a171ad73f86643afb1444248c0d0bc8b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b7d5b9696b4046b522c880c748fe6c for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:29:56.211088 25259 raft_consensus.cc:2396] T 083a9351f8ee4a44a86cb2d42f1ec558 P 45b7d5b9696b4046b522c880c748fe6c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5bf90e514781418da921c046aa5d9ced for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:56.211171 27502 raft_consensus.cc:2726] T 8423ff21d93741cdab3093ea454bb532 P 45b7d5b9696b4046b522c880c748fe6c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240502 21:29:56.211515 24855 raft_consensus.cc:2396] T 083a9351f8ee4a44a86cb2d42f1ec558 P a171ad73f86643afb1444248c0d0bc8b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5bf90e514781418da921c046aa5d9ced for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:29:56.211666 25191 leader_election.cc:302] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45b7d5b9696b4046b522c880c748fe6c; no voters: 64d97dd1674d43ccadd7b4a38eeff557, a171ad73f86643afb1444248c0d0bc8b
I20240502 21:29:56.212697 25058 leader_election.cc:302] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced; no voters: 45b7d5b9696b4046b522c880c748fe6c, a171ad73f86643afb1444248c0d0bc8b
I20240502 21:29:56.213212 27414 raft_consensus.cc:2726] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240502 21:29:56.214743 27502 raft_consensus.cc:2726] T 48e7c6534b544336ae216d3200f4f01f P 45b7d5b9696b4046b522c880c748fe6c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240502 21:29:56.215016 24990 raft_consensus.cc:2454] T 87459ddf48034b28b7b3e0d0d7de91f1 P 64d97dd1674d43ccadd7b4a38eeff557 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf90e514781418da921c046aa5d9ced in term 6.
I20240502 21:29:56.215612 25058 leader_election.cc:302] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5bf90e514781418da921c046aa5d9ced, 64d97dd1674d43ccadd7b4a38eeff557; no voters: 45b7d5b9696b4046b522c880c748fe6c
I20240502 21:29:56.216149 27414 raft_consensus.cc:2781] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:29:56.216311 27414 raft_consensus.cc:686] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [term 6 LEADER]: Becoming Leader. State: Replica: 5bf90e514781418da921c046aa5d9ced, State: Running, Role: LEADER
I20240502 21:29:56.216640 27414 consensus_queue.cc:237] T 87459ddf48034b28b7b3e0d0d7de91f1 P 5bf90e514781418da921c046aa5d9ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } }
I20240502 21:29:56.223284 25124 leader_election.cc:288] T 083a9351f8ee4a44a86cb2d42f1ec558 P 5bf90e514781418da921c046aa5d9ced [CANDIDATE]: Term 7 election: Requested vote from peers 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68:43099), a171ad73f86643afb1444248c0d0bc8b (127.29.82.65:33021)
I20240502 21:29:56.224843 25121 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d643c15afbec4933a0d0a145f17f703c"
dest_uuid: "5bf90e514781418da921c046aa5d9ced"
from {username='slave'} at 127.0.0.1:52128
I20240502 21:29:56.225100 25121 raft_consensus.cc:482] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:56.225266 25121 raft_consensus.cc:3037] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:29:56.236440 25121 raft_consensus.cc:504] T d643c15afbec4933a0d0a145f17f703c P 5bf90e514781418da921c046aa5d9ced [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:56.238193 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d643c15afbec4933a0d0a145f17f703c" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "45b7d5b9696b4046b522c880c748fe6c"
I20240502 21:29:56.238476 25259 raft_consensus.cc:3032] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:29:56.238601 25259 raft_consensus.cc:729] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 45b7d5b9696b4046b522c880c748fe6c, State: Running, Role: LEADER
I20240502 21:29:56.238965 25259 consensus_queue.cc:260] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } }
I20240502 21:29:56.239391 25259 raft_consensus.cc:3037] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:29:56.243291 25259 raft_consensus.cc:2396] T d643c15afbec4933a0d0a145f17f703c P 45b7d5b9696b4046b522c880c748fe6c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5bf90e514781418da921c046aa5d9ced for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240502 21:29:56.246984 24711 catalog_manager.cc:5484] T 6f25dfc6b0c04d878cd3db58d37cddec P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 4 to 6, leader changed from 5bf90e514781418da921c046aa5d9ced (127.29.82.67) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 6 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d97dd1674d43ccadd7b4a38eeff557" member_type: VOTER last_known_addr { host: "127.29.82.66" port: 39721 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67" port: 37875 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } }
I20240502 21:29:56.253754 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d643c15afbec4933a0d0a145f17f703c" candidate_uuid: "5bf90e514781418da921c046aa5d9ced" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "64d97dd1674d43ccadd7b4a38eeff557"
I20240502 21:29:56.254050 24990 raft_consensus.cc:3037] T d643c15afbec4933a0d0a145f17f703c P 64d97dd1674d43ccadd7b4a38eeff557 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:29:56.257141 24707 catalog_manager.cc:5484] T 89b06cd6b69d41d0b49ad40f605a3e16 P 45b7d5b9696b4046b522c880c748fe6c reported cstate change: term changed from 5 to 6, leader changed from a171ad73f86643afb1444248c0d0bc8b (127.29.82.65) to 45b7d5b9696b4046b522c880c748fe6c (127.29.82.68). New cstate: current_term: 6 leader_uuid: "45b7d5b9696b4046b522c880c748fe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a171ad73f86643afb1444248c0d0bc8b" member_type: VOTER last_known_addr { host: "127.29.82.65" port: 33021 } } peers { permanent_uuid: "45b7d5b9696b4046b522c880c748fe6c" member_type: VOTER last_known_addr { host: "127.29.82.68" port: 43099 } } peers { permanent_uuid: "5bf90e514781418da921c046aa5d9ced" member_type: VOTER last_known_addr { host: "127.29.82.67"